Convert Region 25 (MIBOR / Indy) to new updater

[Battler45]

George,

The first project on which you'll be working will be to convert an existing MLS data feed to the new updater system.

Vihar will be able to provide you with the technical details for installing GitHub and downloading a local copy of the current code base.

Vihar, will you also please provide George with whatever server and / or database access he needs for this initial project? Please share with him any necessary logins and create any new users as necessary. Also, I think that converting region 25 will be a good place to start because it shares the same MLS data provider (Marketlinx) as the one you have already converted (SoCal MLS). You can find my existing code in:

C:\Inetpub\mls4.sierrainteractivemls.com\admin\mibor

on DEDQ61

The login credentials to the RETS server are:

• UserID: 31041rets

• Password: BENJ1001

• UserAgent: sierrainteract

• Password: h1ghplateau

RETS URL: http://inr.rets.interealty.com/Login.asmx/Login

As you'll see when you examine my existing code, I was using the sysid name for the database fields rather than the long names for the fields (which were being used for SoCal MLS). I did this because I had a lot of difficulty with the RETS Connector software that I was using to download the daily data files. To help translate between the two, I've attached a spreadsheet provided to me by MIBOR (which stands for Metropolitan Indianapolis Board of Realtors) that cross-references the long field names with their numeric IDs.

Please let me know if you have any questions.

Thanks.

Posted by Peskoe Ben(unfuddle username: bpeskoe)

[Battler45]

George,

Forgot to add - you can check the database schema by remoting to the following machine:

IP: 75.103.119.195

User Name: vshah

Password: ViharShah$123

Pls note that this is development DB, and, has almost identical schema to the live DB. There are total five DBs on this server: sierrainteractive-1 to 4, and sierrainteractive-updater. Pls note that currently sierrainteractive-3 is being used by the live sites so pls make sure that you don't change anything in that database. Ideally, you would only be needing to refer to the sierrainteractive-4 and sierrainteractive-updater DBs.

Pls let me know in case of any issues.

Thanks. Posted by Shah Vihar(unfuddle username: vshah)

[Battler45]

Vihar,

I think it is fine idea to host images on 3rd-party service.

Also I want to notice that MIBOR updater is not completed as I didn't test open-houses updating.

Thanks Posted by Molchanov George(unfuddle username: george)

[Battler45]

Vihar & George,

I just realized that we will, of course, need to re-load all photos because the photos will all be located on a different server, and the folder structure used by the new updater is different than that of the current updater.

Vihar, the problem with this is that we can't just delete the existing region 25 listings from the database, because all of our lead records reference the IDs of these listings in the database. Really, all we need to do is update the MLSPhotos records for each listing -- everything else can stay the same, right?

Do you have any thoughts on what might be the best way to make that transition?

Thanks Posted by Peskoe Ben(unfuddle username: bpeskoe)

[Battler45]

Ben,

Vihang has restored the most recent sierrainteractive-1 DB on dev. DB server. Pls note that he has also updated the dev. front-end sites to point to dev DB for displaying content-pages.

Thanks.

Posted by Shah Vihar(unfuddle username: vshah)

[Battler45]

Ben,

Yes, I made changes within the Inetpub-LIVE folder. To be specific, I wanted to check the old list / detail pages for photos. In doing that, I made changes so that photo urls were invalid. I then checked those pages expecting them to not show photos but photos were displayed correctly. Perhaps it may have something to do with caching; strangely though photos were not rendered in hometoindy list page. I'll spend more time on this later today.

Thanks Posted by Shah Vihar(unfuddle username: vshah)

[Battler45]

George,

I've set you up so that you should be receiving updates on this task / thread -- please confirm. This thread contains the initial details you will need to get started on your first task, and Vihar can provide you with the background, technical details, etc. that you will need to get started.

Please let me know if you have any questions.

Thanks. Posted by Peskoe Ben(unfuddle username: bpeskoe)

[Battler45]

Vihar, What do you mean by MIBOR meta-data? I'm using T5_field_info_RETS__Updated01-17-09.xls document attached in first post. Or you want to see how I perform this mapping (so I can give my code of "Parse.." function). Thanks Posted by Molchanov George(unfuddle username: george)

[Battler45]

George,

Yes, it appears I made a mistake. I thought that we need to pull both Active and Pending listings for the MIBOR feed.

We only need to pull the Active listings.

Can you please update the MIBOR updater accordingly, and then run an update that will remove all Pending listings currently in the database?

Thanks,

Ben Posted by Peskoe Ben(unfuddle username: bpeskoe)

[Battler45]

Vihar,

Populating SourceUniqueID into current listings will not be a problem. I will re-run the MIBOR update for today on the old updater and modify it to include this field for all listings. I will let you and George know as soon as that is done.

Thanks. Posted by Peskoe Ben(unfuddle username: bpeskoe)

[Battler45]

George,

You can find the example for working with RETS meta-data here: http://www.crt.realtors.org/projects/rets/librets/documentation/devguide/. The 3rd point, "Working with metadata" provides some examples of downloading RETS server metadata. This metadata can then be used as reference for mapping MIBOR fields to our database fields.

Thanks. Posted by Shah Vihar(unfuddle username: vshah)

[Battler45]

George,

I've made changes to the image-processing code to generate large images in 640x640 dimensions. I've pushed this change to the master branch. Can you pls fetch the latest code and update your local code?

This just for your info - I've created a new branch, production in the updater git repository. This branch will contain code that is currently used for production updater service. That is, it won't contain any development specific code. I'll maintain this branch to make sure that development specific changes are not pushed to the production service. Pls continue pushing your changes to the master branch (which would contain all development specific changes like you are making for implementing MIBOR updater).

As for photo-download/process testing - I would suggest you to first test it on your local machine (if you haven't already). For this, pls set the LastUpdateTimeStamp for photo-step (16) to current date. This will make sure that only most recent photos are downloaded/processed. Once you've verified the changes, pls test the MIBOR updater on the development server. If everything is working as expected, we can then download all records / photos (by setting the LastUpdateTimeStamp value to null for all steps). Pls let me know your thoughts.

Thanks.

Posted by Shah Vihar(unfuddle username: vshah)

[Battler45]

George,

To further clarify, the existing updater for mibor code will probably be confusing to you. It references the sys IDs of each database field, rather than the long field names, which is what you will be using when you query the RETS server using libRETS (I believe). You can use the T5 Field Info RETS Updated spreadsheet attached to the first comment in this thread to help you see which sys IDs match to which long field names.

In bringing any new data feed in line with the new updater system, the most difficult part will always be to determine how the fields from the host RETS database should be mapped into the fields of our own database.

In certain cases, such as when dealing with City, State, and Zip Code, these are straightforward.

However, in dealing with other types of listing data, such as interior and exterior features, lot features, etc., this can be more complicated. So, in working this out, referring to the existing SoCal MLS updater mappings (region 10) may also be quite helpful for you, because that region will (I believe) contain many of the same field names and will share many similar mappings.

Please let me know if you have any questions.

Thanks. Posted by Peskoe Ben(unfuddle username: bpeskoe)

[Battler45]

George,

In implementing new updater system for region-25, you might want to refer to the existing updater system implementation (especially for field mappings). I've attached a zip file containing the source-code for the same. It contains the following folders:

mibor - contains code specific to updater-system for region 25. The "mls_autoupdates.asp" file contains the entry-point; that is, the programming in this file will initiate the updates for region-25. It will call appropriate functions like UpdateListings, RemoveOldListings, UpdatePhotos etc... for performing various update steps. Some of these functions could be defined in the mls_functions.asp file present under this folder. And, rest of the functions would be defined in the common code.

res/includes - contains code common to all updater-systems. The mls_functions.asp file for example defines InsertUpdateMLSListing function, which is called to add/update listing records.

---

I'm not sure if you had a chance to work with libRETS library. If you haven't, I would recommend first creating a sample project that uses the libRETS library. Pls use the above RETS credentials for connecting to the MIBOR RETS server. Ideally, you should first download the metadata from this server. That would help you in mapping listing fields.

Pls note that the MIBOR region data is stored in the sierrainteractive-1 database. You can copy the latest back up of the database (development) from the following FTP location:

Host: 75.103.116.57 User-name: george_dev_db Password: GeorgeM$123

Since the development DB is has restricted access, pls restore the above back-up on your local DB and configure the updater code to work with your local DB.

Pls let me know if you have any queries.

Thanks.

Posted by Shah Vihar(unfuddle username: vshah)

[Battler45]

George & Vihar,

RE: #69, I've contacted MIBOR representatives to see if they can give us access to the requested resources. I will let you know.

Thanks. Posted by Peskoe Ben(unfuddle username: bpeskoe)

[Battler45]

George,

Sorry for the delayed response -- yes, you are correct that for the initial data load, we will need to import all photos new from the server (this process will likely take a couple of days).

I have been talking with Vihar for the past couple of days, and we are going to change the max dimensions of the large size of the imported photos. Up to now, we have allowed a maximum width or height of 550px, and going forward we are going to allow a max width or height of 640px. Vihar may have already made this change in the code, but please confirm with him before you begin the photo download.

Thanks.

Ben Posted by Peskoe Ben(unfuddle username: bpeskoe)

[Battler45]

George,

Sorry for not being specific. I actually meant the files present under the Updater\Sierra.Updater.Database folder. I believe you've already made changes to the "scheama changes.sql" file to associate steps for MIBOR region. Pls add the new field statement to that file.

Thanks. Posted by Shah Vihar(unfuddle username: vshah)

[Battler45]

Please place further comments into Basecamp, here:

https://basecamp.com/1767592/projects/353414-074-convert-region

[Battler45]

Vihar,

I reviewed the changes, all is fine

Thanks Posted by Molchanov George(unfuddle username: george)

[Battler45]

Ben, Vihar,

I have some questions regarding outstanding issues (geocodes and open houses details).

1. Geocodes. It looks like old updater doesn't get latitude and longtitude from MIBOR response. Isn't? I found in meta data some fields which probably can be used for this - DrivingLatitude and DrivingLongitude but I'm not sure. Anyway in my tests all MIBOR records don't return values for these fields. Also I've added functionality of geocode processing (similar to SoCal functionality) and it works.

2. Open-house updating. Can it be that MIBOR's user doesn't have some rights to get open house data? I've tried to get such data and I received error "User does not have access to Resource named OpenHouse".

Thanks Posted by Molchanov George(unfuddle username: george)

[Battler45]

Ben,

Re. #68 - old and new updaters call the same stored-procedure, spUpdateMLSPhoto for updating a photo-record. So perhaps that sp can be updated to set changed-date in the DateAdded field. Pls let me know if this sound fine. I'll update the sp.

Thanks. Posted by Shah Vihar(unfuddle username: vshah)

[Battler45]

Vihar,

On the RealEstat3000.com site, the home page carousel does reference /res/includes/listings-carousel-2.asp, but within that file is contained a reference to /property-search/widgets/carousel.asp and /property-search/widgets/carousel-close.asp. So, as long as the carousel on RealEstate3000.com home page works, I think we should be fine.

And I am fairly certain that carousel-listings.asp is not used. I had added that file originally, but at a later point realized that the display portion of each carousel is site specific, and so that is when I created the carousel.asp file, but I must have forgotten to remove the carousel-listings.asp file.

Thanks. Posted by Peskoe Ben(unfuddle username: bpeskoe)

[Battler45]

Vihar / George:

RE: #52

Unfortunately, the performance of the existing photos server (DEDQ61) is not at an acceptable level, and so we will need to move all of those photos, either to the new cloud server (where the SoCal photos are) or to a 3rd party service like Amazon or Azure.

The DEDQ61 server is a very old server with an under-powered hard drive, and so at peak times the photos are very slow to load. My goal is that, once we are able to covnert all existing regions over to the new updater system, we will be able to retire the DEDQ61 server.

I do agree that the best long-term approach would be to host the photos on a 3rd-party service, either Amazon or Azure. Do you have any preference toward one or the other. I am really not sure of the technical details of using such a service in combination with our service, and I'm also not sure how to determine the cost. Is that something you could help me understand?

Perhaps we could use MIBOR as a test case for moving photos over to one of these services, with the goal of eventually moving all photos over.

The one issue I have had thus far which has made it difficult to use a CDN is photo expiration. Currently, each listing has a set of photos which gets distributed out via a CDN powered by Level 3. However, because agents change photos for listings on a regular basis, and these updated photos often have the same filename as the original, I have not found a good way to "expire" the old photos when they have been updated, other than to set the expire (TTL) value to 1 day for all photos.

This means that all photos are re-loaded into the CDN once per day no matter what, and creates a much heavier load on the server than there needs to be.

Do you have any thoughts on how this might be improved? My only idea is that when new (replacement) photos are received for an existing listing, they should always be given a new filename. This would eliminate the above problem and ensure that the newer photos are always displayed for every listing, and it would also allow us to cache all photos on the CDN for a much longer time (30 days).

If we do implement something like the above, we just need to make sure that the proper order for the photos is preserved using the OrderID field in the MLSPhotos table.

I agree that this is a pressing issue and so if you could continue to split your time between this topic and the clustering issues, that would be great.

Thanks. Posted by Peskoe Ben(unfuddle username: bpeskoe)

[Battler45]

George,

I tested the latest changes. The ListTypeDescrip value is correctly populated now but the ListType value doesn't reflect the type of listing. It should be 1 for residential, 4 for land / lots and 5 for multi-family. Currently, it set to 1 for all types of listings. Pls make necessary change.

Also, I think we may need to change the way SourceUniqueID field's value is set. Currently the ListingKey (150) value is assigned to this field. I think this value may not be unique in the MIBOR RETS server. Can you pls look into assigning "sysid" value to the SoruceUniqueID field? The SourceUniqueID field should store the unique value, which can be used to identify record in the MIBOR database (kind of primary-key). This value is used mainly in checking physically removed listings (In the PurgeListing step (18)).

Thanks. Posted by Shah Vihar(unfuddle username: vshah)

[Battler45]

Ben,

I've added a couple of new functions at the bottom of Inetpub_LIVE\Sites\MLS_Tools_Integrated\res\includes\functions.asp. These functions are used for forming photo-urls based on the PhotoServer value. In order to test these functions I had switched the condition to generate photo-urls based on the new format.

That said, I realize that I had overlooked the fact that several regions don't have PhotoPath value, and, therefore, photo-url format would be the same regardless of the PhotoServer value. This would have been the case for schulerbauer as photos for regions 3 / 4 don't have PhotoPath values. I think we are fine in this case. My apologies for the oversight.

Thanks. Posted by Shah Vihar(unfuddle username: vshah)

[Battler45]

George,

I've modified the MIBOR updater to ignore listings with "Public Internet" set to "No". I've pushed those changes and planning to run a full update of all Indy listings in the next 15 minutes. If you happen to check this before that can you pls confirm the changes?

Ben - removing the existing listings with the "Public Internet" set to "No" would be a bit tricky in that those listings are already added to the MLS table, and there is no other way to remove them other than removing those MLS records through the backend. I was thinking that it would be fine to execute the spRemoveListingByMLSNumber stored-procedure manually for all such listings. That is, after the full update is completed today, the DateLastUpdate date of all active listings having "Public Internet" set "Yes" would have changed. The DateLastUpdate field of the other listings, most likely having "Public Internet" set "No", would remain unchanged and so it would be fine to remove such listings by executing the spRemoveListingByMLSNumber stored-procedure manually. Pls let me know if you see any issue with this.

Thanks.

Posted by Shah Vihar(unfuddle username: vshah)

[Battler45]

George,

Yes, the specified RETS servers are live. I think it should be ok to connect to those servers for testing purpose.

I understand that you want to test the new updater system - the SoCal updater system to be specific. To do so, you can run the WindowsServiceTest console application. The UpdateProcessor.cs file in this project contains the entry-point function, PerformUpdatesForRegion. This function will execute appropriate updater steps (depending on the steps supported for a region - as defined in the Updater_MLSRegionUpdateSteps table). If you want to test a specific step, pls comment out appropriate code in the PerformUpdatesForRegion function.

Also, you would need to change certain configuration parameters defined in the app.config file. Some of these parameters are:

• Connection-Strings: updaterConnString & connString1 - pls set these to point to your local db server.
• PhotoRootFolder: this is the path where photos would be downloaded from the MLS server
• LogEmails: true / false. If set to true, alert-emails would not be sent to actual email addresses; they would instead be logged in the log file
• TestEmailAddress: if the LogEmails parameter is set to true and this parameter's value is set then emails would be sent to this test email address as well.

Pls let me know if you need additional info.

Thanks.

Posted by Shah Vihar(unfuddle username: vshah)

[Battler45]

George,

Through Librets, there is a way to pull the system meta-data, which is an outline of the table structures of all tables -- kind of like the database schema. So, it gives you all of the available fields for the Residential property type, what data type each field is, the length, etc. And it does this for each table in the RETS database.

I'm not sure how you pull the meta-data using librets, but if you search the librets documentation for "metadata" or "meta-data" I am sure there is an easy way to do it.

This is similar to the spreadsheet I attached in the first email, but a bit different in that it is more complete and more up-to-date, and so it would probably be preferable to use this as opposed to the spreadsheet.

RE: #11 above:

1 - These are good questions. Each data feed is, unfortunately, different. In the case of the MIBOR feed, they only use one field for the full agent name ("Listing Agent Name"), rather than separate fields for the first and last name, so you will need to use string functions to parse the full name into a first and last name.

2 - Yes, it will usually be the case that not all available fields in the data feed will be mapped. This is the most difficult part of setting up a new data feed -- determining which fields should be mapped, and which should not. Often there will be multiple fields with similar names, and we will need to determine which one contains the data we are actually looking for.

There are certain fields which all feeds will contain (using different field names) and that we will always want to use, such as agent name, listing price, and MLS #. I will try to prepare a list of these to send over to you.

Then, there are fields available in some feeds but not others, such as different types of interior and exterior features, etc. The difficult part is determining which of these fields need to be included. I will help you with this now, and over time you will get a better sense of this.

3 - Some of our fields (mainly those beginning with "Feature" such as "FeaturesExterior") do contain composites of multiple fields from the data feed. There are no fixed rules here, and so while FeaturesExterior might combine 4 different data fields in SoCal MLS, it might combine 5 different fields in MIBOR. This needs to be worked out on a case-by-case basis, following common-sense guidelines which will become more clear as you gain experience working with these systems.

To help you start, I will try to prepare a list of which fields in the data feed should be mapped to which fields in our database and will send that over to you later today, along with a general explanation of the type of data each composite field should include.

Thanks. Posted by Peskoe Ben(unfuddle username: bpeskoe)

[Battler45]

Ben,

I've made changes to the production and dev DBs to add a new column, PhotoServer (bit) to the MLSPhotos table. When 0, it will indicate photos hosted on the old server, and 1 will indicate photos hosted on the cloud server. Currently its value is set to 1 for all records in the sierrainteractive-4 DB and 0 for all other records.

As mentioned in the above comment, the photo-migration app is working fine so we can now begin working on transitioning the MIBOR updater and its photos to the cloud server. I'll test the MIBOR updater on development server tomorrow; will update you on the same afterwards.

Also, we can now update the programming to build photo-urls to use the PhotoServer value. I think it would be a good idea to create a common function, which can then be called from all pages for displaying photos. My understanding is that the following pages will need to be updated:

• list
• map
• detail
• detail2
• search-application
• content widget

Pls let me know your thoughts. I'll also update the new CAA accordingly.

Thanks.

Posted by Shah Vihar(unfuddle username: vshah)

[Battler45]

Vihar,

Unfortunately I can not remote into 208.106.250.174 to update database with my scripts - maximum number of allowed connection is exceeded. Also I have again question about git - I've run git fetch command but I don't see any changes (there should be changed ImageHelper.cs and Constants.cs). What I have to do more?

Thanks Posted by Molchanov George(unfuddle username: george)

[Battler45]

Vihar,

I've fixed issues with ListType and SourceUniqueID.

Thanks Posted by Molchanov George(unfuddle username: george)

[Battler45]

George & Vihar,

Vihar, would you be able to modify the connection strings on the dev version of the search tools to pull from the development DB for sierrainteractive-1 sites?

Then, George, you will be able to test your work using the http://hometoindy.sierrainteractivedev.com site -- what you see there will need to match what you see on http://www.hometoindy.com when viewing details for the same listing.

This should allow you to test your implementation until the data in the development database (using your updater) matches exactly with the data in the live database (currently running on the old updater).

Vihar, I don't know if there is a way to allow George to query the region 25 records on the live DB for comparison purposes -- perhaps just viewing the data on the live site would be sufficient?

Thanks Posted by Peskoe Ben(unfuddle username: bpeskoe)

[Battler45]

George & Vihar,

I have spoken with the staff at MIBOR and they are considering granting us access to the requested resources (History, Deleted) but they have not done so yet, and they have not yet committed to do so.

So, for the time being, I think we need to develop an alternative way to remove expired listings from our database.

The only way I can see to do this would be to pull a complete list of all active listings each day -- either by sysID, or MLS #, or both -- and use that list to compare against the listings database and identify the listings which need to be removed.

I'm not sure how timeconsuming it will be to pull that complete list of all listings each day, but hopefully since you are just pulling a single piece of data, it should not be too bad.

Please let me know if you have any questions.

Thanks. Posted by Peskoe Ben(unfuddle username: bpeskoe)

[Battler45]

Vihang,

I've already done this -- updated the dev sites on DEDN173 to pull content from the live db rather than the development db (I need to work on a new site today).

Thanks Posted by Peskoe Ben(unfuddle username: bpeskoe)

[Battler45]

Ben,

I believe George hasn't yet run the MIBOR updater on the dev DB so it should be fine. I'll confirm with George and restore the back up based on his response.

George - can you pls confirm if it would be alright to go ahead with the restore of sierrainteractive-1 DB?

Thanks.

Posted by Shah Vihar(unfuddle username: vshah)

[Battler45]

Ben,

Developing a small application for data comparison sounds good. The only concern I've is that in using this application, the old and new updater would have to be synchronized. That is, the tests would have to be carried out only after the old updater has run; otherwise the comparison application won't be much of a use. Could you pls confirm the time when the MIBOR updates are completed (through the old updater)? George can then schedule the new updaters to run around that time, and, then can use comparison application to verify correctness of data. Pls let me know your thoughts on this.

Thanks. Posted by Shah Vihar(unfuddle username: vshah)

[Battler45]

George,

RE: #79 - I've reviewed the photo-migration application code. It looks very good. I'll test the application on my local machine. I'll let you know if I run into any issues. As for your questions:

1 -- If a photo cannot be processed for some reason, we should let it remain in the database. The application will log exceptions so that should help us in reviewing such photos. Since the PhotoServer value of such photo-records would be 0, they would continue to be served from the old server (assuming image-files exist and are in valid format)

2 -- As for the database changes (new field and stored-procedures) - since those changes are related to the updater system, pls add them in the "schema changes.sql" and "stored procedures.sql" files respectively.

Thanks. Posted by Shah Vihar(unfuddle username: vshah)

[Battler45]

Vihar,

Sounds great, thanks.
Posted by Peskoe Ben(unfuddle username: bpeskoe)

[Battler45]

George,

The HomeToIndy.com client phoned today to report that her board has released a new class of data -- data for "Past Solds", or listings that have sold and are no longer on the market.

Can you pull the metadata for the MIBOR / Indy feed and see if there are new resources in the feed which would allow us to acccess that data?

Thanks. Posted by Peskoe Ben(unfuddle username: bpeskoe)

[Battler45]

Ben,

RE: #66: I've restarted the DEDN173 server but I'm now unable to remote into it. The server appears to have restarted correctly as I could access the CAA website. Could you pls look into the same?

As for the size of SoCal photos, it is definitely lot more when compared to other regions. I verified that all these photos are associated with active listings. Moreover, the ratio of photos added to photos removed is more than 2 majority of the times so this size in all likelihood will increase.

Thanks.

Posted by Shah Vihar(unfuddle username: vshah)

[Battler45]

Ben,

Thanks. The changes to .htaccess have now taken effect. Pls note that I've reset the passwords for the 'n3wt0n' and 's13rra' for consistency.

Pls let me know your inputs on comment #99 when you have chance.

Thanks. Posted by Shah Vihar(unfuddle username: vshah)

[Battler45]

Vihar,

Yes, there is some sort of bug / problem with ISAPI Rewrite that changes to an .htaccess file do not always take effect once applied. In such cases, it is necessary to restart IIS services on the server in order to get the changes to take effect. I've gone ahead and done that on DEDN173.

The Weir Properties dev site (and all other sites pointing to development DBs) are no longer working due to changes Crystaltech had to make to usernames / passwords. In completing their work, they said, "I've uninstalled SQL from the OS volume and then reinstalled it to your DATA volume. Please note that you will need to update the passwords for the 'n3wt0n' and 's13rra' users as we did not have a PW reference for them, so I used the password listed for your primary login user that we have on file in the control center." I believe the password they are referring to is the same one we use to RDP into the server, but it seems like these should be reset to what they were previously for consistency's sake.

Thanks. Posted by Peskoe Ben(unfuddle username: bpeskoe)

[Battler45]

Vihar, I got RETS meta-data with using of libRETS code samples. Attached are archive with meta-data in text and xml format. In my opinion spreadsheet T5field_info_RETS_Updated01-17-09.xls looks more complete. For example ResidentialProperty object: Field #3: in spreadsheet it has name "Acres", in meta-data isn't any standard name for this field ([] - it means standard name is missed). I thought I can use this value to fill field Acreage of Property object. Is my assumption wrong? Posted by Molchanov George(unfuddle username: george)

[Battler45]

George,

Just checking in to see how things are coming -- were my last comments helpful, and have you been able to make further progress?

Thanks. Posted by Peskoe Ben(unfuddle username: bpeskoe)

[Battler45]

Vihar,

I thought scripts under Updater\Sierra.Updater.Database folder are intended only for sierrainteractive-updater database. Isn't? And my script has to be run on sierrainteractive-n databases.

Also I attach script here for your testing while it is not committed.

Thanks Posted by Molchanov George(unfuddle username: george)

[Battler45]

Vihar,

I've committed code of photo-migration tool. You can find two new projects Sierra.Updater.PhotoMigrating (classes to perform migrating actions) and Sierra.Updater.PhotoMigratingApp (console application).

Now there is implemented simple writer of images into file system (new folders' structure). It can be easily replaced with any other writer (for example with uploader to some cloud system).

I have two questions

1. What should be behaviour of application when some photo can not be processed? Should we remove such record from database completely or we have to mark it as problem (it means we need one more field in database)?

2. How I have to commit database changes? My changes are

• script to add PhotoServer field in table MLSPhoto;

• two stored procedures in database with table MLSPhoto.

Thanks Posted by Molchanov George(unfuddle username: george)

[Battler45]

Vihar,

I completed local testing. As I understand I have to add the appropriate records into table Updater_MLSRegionUpdateSteps. What should be set for field [LastUpdateTimeStamp], i.e. do we need to receive all records from REST server or not? And also I'm ready to perform testing on dev DB as you wrote in note 25.

Thanks Posted by Molchanov George(unfuddle username: george)

[Battler45]

Vihar,

Yes, I believe the issue is caused by several new sites having been added since the development db was last updated. Because adding new sites involves addition of records into multiple tables, it might be best to restore sierrainteractive-1 on the development DB. However, would this impact the work that George is doing and / or has already done? (Erasing test data that he has imported.)

I have created a backup of sierrainteractive-1 called sierrainteractive-1_2011-08-29.bak and downloaded that onto DEDN173 in C:\DB Backups. Please let me know if you have any trouble with that.

Your approach for assisting George to test the updater for region 25 sounds good.

Thanks. Posted by Peskoe Ben(unfuddle username: bpeskoe)

[Battler45]

George,

I tested the MIBOR updater on local machine. Overall, it appears to be working fine. There are a few points that might require changes:

1 -- Looks like the DrivingLongitude & DrivingLatitude values are not available for majority of the listings so we would have to geo-code all listings through the Geocoder step. In that case, I think, the code, which processes latitude / longitude values (last few lines in the IndianapolisListingsImporterForProperty.cs file) can be removed.

2 -- The ListType and ListTypeDescrip values are not correctly set. The ListType value should reflect the type of listing (i.e. 1 for residential, 4 for land/lot, 5 for multi-family and so on...). The ListTypeDescrip value should reflect sub-type (i.e. condominium, residential, single lot and so on...)

Ben - I checked the meta-data for the MIBOR region but couldn't find any suitable field, which could be holding sub-type description. I've attached a zip file containing meta-data / actual data for your reference. Pls let me know the field, which can be used for populating ListTypeDescrip value.

3 -- Per the current implementation, importer downloads all listings regardless of their status. Ideally, it should only process listings having status Active / Pending. There is a field, Status (134) which can be used for retrieving Active / Pending listings only.

Ben - Strangely though when I queried the RETS server to get listings filtered by change-date, it returned active / pending listings only. Perhaps listings with other status might not have been changed. Per the meta-data, a listing can have one of the following statuses: Active, Expired, Leased, Pending, Released, Sold or Withdrawn. Do you think it would be fine to ignore the listing status?

4 -- In the live database, each section in the Featured* field is enclosed within

. For example,

FeaturesBasement -

Basement: [Actual Values]

FeaturesConstruction -

Construction Stage: [Actual Values]

Construction Type: [Actual Values]

Construction Status: [Actual Values]

Pls change the MIBOR updater to set Features* values accordingly.

5 -- Per the current implementation of photo-downloader, if a listing already has one or more photos in the database then all those photos are removed and re-downloaded. I think this might increase the time to download / process all photos. I guess it should be alright to remove all photos only if the number of photos on the RETS server is less compared to the number of photos in the database. Pls let me know if you see any issue with this.

6 -- The only missing step (other than the OpenHouse step) is removal / purging of old listings. As and when listings are sold / withdrawn, they are made unavailable from the REST server (either physically or by changing their status). We need to take this into account in synchronizing RETS data to our database. So an additional step will need to be implemented (in line with the SoCal listing purger step (18)), which will process such listings and remove those from our database. Can you pls review the SoCalListingListingPurger class and implement similar step for MIBOR region?

Pls let me know if you have any queries.

Thanks.

Posted by Shah Vihar(unfuddle username: vshah)

[Battler45]

Ben,

I've made changes in the new CAA and new updater systems to use the PhotoServer value in building photo urls (not published yet). I'm now working on making similar changes in the front-end sites programming. I'm looking into updating the dev. version of search-tools / content pages for the same. I'll update the live sites tomorrow (during your overnight period).

Thanks. Posted by Shah Vihar(unfuddle username: vshah)

[Battler45]

Vihang,

Since we need to test content for the sites in development and we'll be entering that content in the live DB, could you restore the development sites to pull content from the live DB rather than the development DB?

Thanks. Posted by Peskoe Ben(unfuddle username: bpeskoe)