Convert Region 25 (MIBOR / Indy) to new updater

[Battler45]

George,

The first project on which you'll be working will be to convert an existing MLS data feed to the new updater system.

Vihar will be able to provide you with the technical details for installing GitHub and downloading a local copy of the current code base.

Vihar, will you also please provide George with whatever server and / or database access he needs for this initial project? Please share with him any necessary logins and create any new users as necessary. Also, I think that converting region 25 will be a good place to start because it shares the same MLS data provider (Marketlinx) as the one you have already converted (SoCal MLS). You can find my existing code in:

C:\Inetpub\mls4.sierrainteractivemls.com\admin\mibor

on DEDQ61

The login credentials to the RETS server are:

• UserID: 31041rets

• Password: BENJ1001

• UserAgent: sierrainteract

• Password: h1ghplateau

RETS URL: http://inr.rets.interealty.com/Login.asmx/Login

As you'll see when you examine my existing code, I was using the sysid name for the database fields rather than the long names for the fields (which were being used for SoCal MLS). I did this because I had a lot of difficulty with the RETS Connector software that I was using to download the daily data files. To help translate between the two, I've attached a spreadsheet provided to me by MIBOR (which stands for Metropolitan Indianapolis Board of Realtors) that cross-references the long field names with their numeric IDs.

Please let me know if you have any questions.

Thanks.

Posted by Peskoe Ben(unfuddle username: bpeskoe)

[Battler45]

George,

I tested the MIBOR updater on local machine. Overall, it appears to be working fine. There are a few points that might require changes:

1 -- Looks like the DrivingLongitude & DrivingLatitude values are not available for majority of the listings so we would have to geo-code all listings through the Geocoder step. In that case, I think, the code, which processes latitude / longitude values (last few lines in the IndianapolisListingsImporterForProperty.cs file) can be removed.

2 -- The ListType and ListTypeDescrip values are not correctly set. The ListType value should reflect the type of listing (i.e. 1 for residential, 4 for land/lot, 5 for multi-family and so on...). The ListTypeDescrip value should reflect sub-type (i.e. condominium, residential, single lot and so on...)

Ben - I checked the meta-data for the MIBOR region but couldn't find any suitable field, which could be holding sub-type description. I've attached a zip file containing meta-data / actual data for your reference. Pls let me know the field, which can be used for populating ListTypeDescrip value.

3 -- Per the current implementation, importer downloads all listings regardless of their status. Ideally, it should only process listings having status Active / Pending. There is a field, Status (134) which can be used for retrieving Active / Pending listings only.

Ben - Strangely though when I queried the RETS server to get listings filtered by change-date, it returned active / pending listings only. Perhaps listings with other status might not have been changed. Per the meta-data, a listing can have one of the following statuses: Active, Expired, Leased, Pending, Released, Sold or Withdrawn. Do you think it would be fine to ignore the listing status?

4 -- In the live database, each section in the Featured* field is enclosed within

. For example,

FeaturesBasement -

Basement: [Actual Values]

FeaturesConstruction -

Construction Stage: [Actual Values]

Construction Type: [Actual Values]

Construction Status: [Actual Values]

Pls change the MIBOR updater to set Features* values accordingly.

5 -- Per the current implementation of photo-downloader, if a listing already has one or more photos in the database then all those photos are removed and re-downloaded. I think this might increase the time to download / process all photos. I guess it should be alright to remove all photos only if the number of photos on the RETS server is less compared to the number of photos in the database. Pls let me know if you see any issue with this.

6 -- The only missing step (other than the OpenHouse step) is removal / purging of old listings. As and when listings are sold / withdrawn, they are made unavailable from the REST server (either physically or by changing their status). We need to take this into account in synchronizing RETS data to our database. So an additional step will need to be implemented (in line with the SoCal listing purger step (18)), which will process such listings and remove those from our database. Can you pls review the SoCalListingListingPurger class and implement similar step for MIBOR region?

Pls let me know if you have any queries.

Thanks.

Posted by Shah Vihar(unfuddle username: vshah)

[Battler45]

George,

Yes, it appears I made a mistake. I thought that we need to pull both Active and Pending listings for the MIBOR feed.

We only need to pull the Active listings.

Can you please update the MIBOR updater accordingly, and then run an update that will remove all Pending listings currently in the database?

Thanks,

Ben Posted by Peskoe Ben(unfuddle username: bpeskoe)

[Battler45]

Vihar,

Could you please give me git reference to the storage with "schema changes.sql" and "stored procedures.sql"? My change are for database with MLSPhoto table but I have now only updater system's git reference.

Thanks Posted by Molchanov George(unfuddle username: george)

[Battler45]

Vihar,

RE: #63 - Your alternate approach does sound good and workable. I guess that appending a querystring parameter to the photo filename does not impact the ability of the browser to render the images? (I've tested, and it does not seem to be a problem.) In that case, this does seem like the best and easiest solution.

I have not yet checked when and how the DateAdded field in the MLSPhotos table is updated, but I will do so tomorrow, and will make sure that the old updater is set to update this field appropriately. Can you check the new updater to make sure it is also updating this value appropriately?

Thanks Posted by Peskoe Ben(unfuddle username: bpeskoe)

[Battler45]

George & Vihar,

Vihar, would you be able to modify the connection strings on the dev version of the search tools to pull from the development DB for sierrainteractive-1 sites?

Then, George, you will be able to test your work using the http://hometoindy.sierrainteractivedev.com site -- what you see there will need to match what you see on http://www.hometoindy.com when viewing details for the same listing.

This should allow you to test your implementation until the data in the development database (using your updater) matches exactly with the data in the live database (currently running on the old updater).

Vihar, I don't know if there is a way to allow George to query the region 25 records on the live DB for comparison purposes -- perhaps just viewing the data on the live site would be sufficient?

Thanks Posted by Peskoe Ben(unfuddle username: bpeskoe)

[Battler45]

Vihar & George,

Thanks, everything appears to have updated correctly and to be working properly now.

George, I do have one more question for the Indy updater -- the contact there says that the Sold data for this region should now be available through the RETS feed. Can you check the RETS metadata for this reason and see if the Sold data is now available as a new class?

Thanks. Posted by Peskoe Ben(unfuddle username: bpeskoe)

[Battler45]

George,

My apologies for the delay in responding you. I've verified the changes for ListType and SourceUniqueID; they are working fine.

Can you pls let me know if you have begun working on the photo-migration application?

Thanks. Posted by Shah Vihar(unfuddle username: vshah)

[Battler45]

Vihar,

I implemented the new approach to delete listings of MIBOR region. It works pretty fast (about 28 thousand records from MIBOR RETS server). All is committed excepting sql script with stored procedure for sierrainteractive-n databases (it is attached).

Thanks Posted by Molchanov George(unfuddle username: george)

[Battler45]

George,

The HomeToIndy.com client phoned today to report that her board has released a new class of data -- data for "Past Solds", or listings that have sold and are no longer on the market.

Can you pull the metadata for the MIBOR / Indy feed and see if there are new resources in the feed which would allow us to acccess that data?

Thanks. Posted by Peskoe Ben(unfuddle username: bpeskoe)

[Battler45]

Vihar,

Populating SourceUniqueID into current listings will not be a problem. I will re-run the MIBOR update for today on the old updater and modify it to include this field for all listings. I will let you and George know as soon as that is done.

Thanks. Posted by Peskoe Ben(unfuddle username: bpeskoe)