search

Battler45 / SierraMigrationToGitHub

Migration from unfuddle to GitHub
0 stars 0 forks source link

Convert Region 25 (MIBOR / Indy) to new updater #38

Open Battler45 opened 4 years ago

Battler45 commented 4 years ago

George,

The first project on which you'll be working will be to convert an existing MLS data feed to the new updater system.

Vihar will be able to provide you with the technical details for installing GitHub and downloading a local copy of the current code base.

Vihar, will you also please provide George with whatever server and / or database access he needs for this initial project? Please share with him any necessary logins and create any new users as necessary. Also, I think that converting region 25 will be a good place to start because it shares the same MLS data provider (Marketlinx) as the one you have already converted (SoCal MLS). You can find my existing code in:

C:\Inetpub\mls4.sierrainteractivemls.com\admin\mibor

on DEDQ61

The login credentials to the RETS server are:

  • UserID: 31041rets

  • Password: BENJ1001

  • UserAgent: sierrainteract

  • Password: h1ghplateau

RETS URL: http://inr.rets.interealty.com/Login.asmx/Login

As you'll see when you examine my existing code, I was using the sysid name for the database fields rather than the long names for the fields (which were being used for SoCal MLS). I did this because I had a lot of difficulty with the RETS Connector software that I was using to download the daily data files. To help translate between the two, I've attached a spreadsheet provided to me by MIBOR (which stands for Metropolitan Indianapolis Board of Realtors) that cross-references the long field names with their numeric IDs.

Please let me know if you have any questions.

Thanks.

Posted by Peskoe Ben(unfuddle username: bpeskoe)

Battler45 commented 4 years ago

Vihar & George,

Thanks, everything appears to have updated correctly and to be working properly now.

George, I do have one more question for the Indy updater -- the contact there says that the Sold data for this region should now be available through the RETS feed. Can you check the RETS metadata for this reason and see if the Sold data is now available as a new class?

Thanks. Posted by Peskoe Ben(unfuddle username: bpeskoe)

Battler45 commented 4 years ago

Vihar, George,

RE: #54

1 - Yes, please do not use Driving Longitude / Driving Latitude values from the MIBOR feed. Instead, use our Geocoding scripts which will geocode listings directly based on property address.

2 - For ListType 1 (Residential) the field is 511: Res/Cnd for ListTypeDescrip. For ListType 4 (Land) the field is 343: Type for ListTypeDescrip. For ListType 5 (Multi-Faimily), the field is 135: List Type for ListTypeDescrip.

3 - Vihar, I believe you are correct and that our access is restricted to only be able to retrieve Active / Pending listings. However, it would still be nice to be able to track listing status if we can using the Status field in the MLS table. (In our front-end application, we may at some point want to display status to end-users.)

4 - Regarding usage of

, that was an attempt on my part to improve the way the listings are displayed on the front end. However, it was a failed experiment, and so it's fine to format the Features* values in the same way that they are formatted in the SoCal MLS data feed. In fact, it would be preferable to keep it the same as the SoCal format for consistency's sake.

5 - I'm not sure about this. In some cases, an agent my replace existing photos with a new set of photos, but the overall number of photos may not have changed. In such cases, it would definitely be necessary to delete all existing photos and re-download the full set to insure that all new photos are properly received.

Please let me know if you have any questions

Thanks Posted by Peskoe Ben(unfuddle username: bpeskoe)

Battler45 commented 4 years ago

Vihar & George,

Can you update me on the status of this? Has the updater completed and have the indicated listings been removed from the site? I do see that the original listing MLS # 21163069 is no longer available, but wanted to confirm before responding to the MIBOR contact.

He also just emailed to request that we verify we are not displaying the address for listings where the "Address yes/no" field is set to "No" -- I believe we are changing the address on these listings to "000 Confidential" but can you double-check that for me as well?

Thanks. Posted by Peskoe Ben(unfuddle username: bpeskoe)

Battler45 commented 4 years ago

Vihar, Thanks for your help. I will check this listing. Regarding composite fields for Residental Property. Now I'm able to fill next composite fields with same headers as for SoCal:

For other composite fields (FeaturesCommunity, FeaturesCondo, FeaturesConstruction, FeaturesDocuments, FeaturesExterior, FeaturesFencing, FeaturesHOA, FeaturesParking, FeaturesRoof, FeaturesUtilities) I can not find MIBOR fields which could be assigned to SoCal headers. Probably some composite fields can be filled with MIBOR data with other headers (for example MIBOR contains "Garage description" field which can be placed into composite field FeaturesParking). But I'm not sure is it legal or not and who should make such decisions. Posted by Molchanov George(unfuddle username: george)

Battler45 commented 4 years ago

George,

The HomeToIndy.com client phoned today to report that her board has released a new class of data -- data for "Past Solds", or listings that have sold and are no longer on the market.

Can you pull the metadata for the MIBOR / Indy feed and see if there are new resources in the feed which would allow us to acccess that data?

Thanks. Posted by Peskoe Ben(unfuddle username: bpeskoe)

Battler45 commented 4 years ago

George,

Through Librets, there is a way to pull the system meta-data, which is an outline of the table structures of all tables -- kind of like the database schema. So, it gives you all of the available fields for the Residential property type, what data type each field is, the length, etc. And it does this for each table in the RETS database.

I'm not sure how you pull the meta-data using librets, but if you search the librets documentation for "metadata" or "meta-data" I am sure there is an easy way to do it.

This is similar to the spreadsheet I attached in the first email, but a bit different in that it is more complete and more up-to-date, and so it would probably be preferable to use this as opposed to the spreadsheet.

RE: #11 above:

1 - These are good questions. Each data feed is, unfortunately, different. In the case of the MIBOR feed, they only use one field for the full agent name ("Listing Agent Name"), rather than separate fields for the first and last name, so you will need to use string functions to parse the full name into a first and last name.

2 - Yes, it will usually be the case that not all available fields in the data feed will be mapped. This is the most difficult part of setting up a new data feed -- determining which fields should be mapped, and which should not. Often there will be multiple fields with similar names, and we will need to determine which one contains the data we are actually looking for.

There are certain fields which all feeds will contain (using different field names) and that we will always want to use, such as agent name, listing price, and MLS #. I will try to prepare a list of these to send over to you.

Then, there are fields available in some feeds but not others, such as different types of interior and exterior features, etc. The difficult part is determining which of these fields need to be included. I will help you with this now, and over time you will get a better sense of this.

3 - Some of our fields (mainly those beginning with "Feature" such as "FeaturesExterior") do contain composites of multiple fields from the data feed. There are no fixed rules here, and so while FeaturesExterior might combine 4 different data fields in SoCal MLS, it might combine 5 different fields in MIBOR. This needs to be worked out on a case-by-case basis, following common-sense guidelines which will become more clear as you gain experience working with these systems.

To help you start, I will try to prepare a list of which fields in the data feed should be mapped to which fields in our database and will send that over to you later today, along with a general explanation of the type of data each composite field should include.

Thanks. Posted by Peskoe Ben(unfuddle username: bpeskoe)

Battler45 commented 4 years ago

Ben,

I've updated dev. version of search-tools to use PhotoServer value in forming photo-url. I've also updated & verified a couple pages of live sites on DEDN173 for the same. I'll complete the remaining changes tomorrow. Several widget pages (\MLS_Tools_Integrated\widgets) have been updated but I'm not sure which pages embed those widgets. Could you pls let me know how I can check the following widgets:

Moreover, several xml feeds have also been updated. Is there any specific way to check the following feeds?

Pls let me know your inputs.

Thanks. Posted by Shah Vihar(unfuddle username: vshah)

Battler45 commented 4 years ago

Vihar,

Yes, that does sound like a good plan, please proceed.

Thanks. Posted by Peskoe Ben(unfuddle username: bpeskoe)

Battler45 commented 4 years ago

George,

Sorry for the delayed response -- yes, you are correct that for the initial data load, we will need to import all photos new from the server (this process will likely take a couple of days).

I have been talking with Vihar for the past couple of days, and we are going to change the max dimensions of the large size of the imported photos. Up to now, we have allowed a maximum width or height of 550px, and going forward we are going to allow a max width or height of 640px. Vihar may have already made this change in the code, but please confirm with him before you begin the photo download.

Thanks.

Ben Posted by Peskoe Ben(unfuddle username: bpeskoe)

Battler45 commented 4 years ago

George,

Sorry for the delay getting back to you on this -- I am working on it now and will have it for you by the end of the evening (my time). In the meantime, if you have more time to work on the project, perhaps you could begin looking into the photo downloads for this region? That (hopefully) should be very similar to the way it is currently being done for SoCal MLS / Region 10.

Thanks. Posted by Peskoe Ben(unfuddle username: bpeskoe)

Battler45 commented 4 years ago

Vihar, I got RETS meta-data with using of libRETS code samples. Attached are archive with meta-data in text and xml format. In my opinion spreadsheet T5field_info_RETS_Updated01-17-09.xls looks more complete. For example ResidentialProperty object: Field #3: in spreadsheet it has name "Acres", in meta-data isn't any standard name for this field ([] - it means standard name is missed). I thought I can use this value to fill field Acreage of Property object. Is my assumption wrong? Posted by Molchanov George(unfuddle username: george)

Battler45 commented 4 years ago

Vihar,

Ok, I understood. Will do it.

Thanks. Posted by Molchanov George(unfuddle username: george)

Battler45 commented 4 years ago

Vihar,

On the RealEstat3000.com site, the home page carousel does reference /res/includes/listings-carousel-2.asp, but within that file is contained a reference to /property-search/widgets/carousel.asp and /property-search/widgets/carousel-close.asp. So, as long as the carousel on RealEstate3000.com home page works, I think we should be fine.

And I am fairly certain that carousel-listings.asp is not used. I had added that file originally, but at a later point realized that the display portion of each carousel is site specific, and so that is when I created the carousel.asp file, but I must have forgotten to remove the carousel-listings.asp file.

Thanks. Posted by Peskoe Ben(unfuddle username: bpeskoe)

Battler45 commented 4 years ago

George,

A few points for your info:

1 -- The SoCal and Indianapolis Geocoder classes were same so I've removed one of them, and, have renamed other as DefaultGeocoder class. It will be used in Geocoding listings for both these regions. In future, if any region's Geocoding step diverges, a new class can be implemented for that region; otherwise we can continue to use the DefaultGeocoder class.

2 -- I noticed that the DateListed value (which represents the data when a property was listed, and is used in calculating DaysOnMarket value) was always set to current date. Apparently, the format of the ListDate field in the RETS was different. I've made changes accordingly. It now correctly sets DateListed and DaysOnMarket values.

3 -- For each region, a couple of addition steps need to be executed viz. Statistics Calculations (it calculates stats like average price / median price etc...) and Generating Email Alerts (sending emails to leads who have opted to be notified when new listings matching their criteria are added). These steps are common; no additional programming is required. We'll just need to associate the region with the corresponding step in the Updater_MLSRegionUpdateSteps table. The step-ids are 21 and 20 respectively. I think the email-alert step (20) can be associated once we have thoroughly tested the MIBOR region updater.

4 -- To make testing of a single region easier, I've added a configurable parameter, MLSRegionToTest in the console-application. When its value is set to one of the region-ids (i.e. 25) it will perform updates for that specific region. When set to -1, it will perform updates for all regions. I think this would be useful as when we convert regions to new updater system.

Thanks.

Posted by Shah Vihar(unfuddle username: vshah)

Battler45 commented 4 years ago

Vihang,

I've already done this -- updated the dev sites on DEDN173 to pull content from the live db rather than the development db (I need to work on a new site today).

Thanks Posted by Peskoe Ben(unfuddle username: bpeskoe)