Convert Region 25 (MIBOR / Indy) to new updater

[Battler45]

George,

The first project on which you'll be working will be to convert an existing MLS data feed to the new updater system.

Vihar will be able to provide you with the technical details for installing GitHub and downloading a local copy of the current code base.

Vihar, will you also please provide George with whatever server and / or database access he needs for this initial project? Please share with him any necessary logins and create any new users as necessary. Also, I think that converting region 25 will be a good place to start because it shares the same MLS data provider (Marketlinx) as the one you have already converted (SoCal MLS). You can find my existing code in:

C:\Inetpub\mls4.sierrainteractivemls.com\admin\mibor

on DEDQ61

The login credentials to the RETS server are:

• UserID: 31041rets

• Password: BENJ1001

• UserAgent: sierrainteract

• Password: h1ghplateau

RETS URL: http://inr.rets.interealty.com/Login.asmx/Login

As you'll see when you examine my existing code, I was using the sysid name for the database fields rather than the long names for the fields (which were being used for SoCal MLS). I did this because I had a lot of difficulty with the RETS Connector software that I was using to download the daily data files. To help translate between the two, I've attached a spreadsheet provided to me by MIBOR (which stands for Metropolitan Indianapolis Board of Realtors) that cross-references the long field names with their numeric IDs.

Please let me know if you have any questions.

Thanks.

Posted by Peskoe Ben(unfuddle username: bpeskoe)

[Battler45]

George,

Forgot to add - you can check the database schema by remoting to the following machine:

IP: 75.103.119.195

User Name: vshah

Password: ViharShah$123

Pls note that this is development DB, and, has almost identical schema to the live DB. There are total five DBs on this server: sierrainteractive-1 to 4, and sierrainteractive-updater. Pls note that currently sierrainteractive-3 is being used by the live sites so pls make sure that you don't change anything in that database. Ideally, you would only be needing to refer to the sierrainteractive-4 and sierrainteractive-updater DBs.

Pls let me know in case of any issues.

Thanks. Posted by Shah Vihar(unfuddle username: vshah)

[Battler45]

George,

I've set you up so that you should be receiving updates on this task / thread -- please confirm. This thread contains the initial details you will need to get started on your first task, and Vihar can provide you with the background, technical details, etc. that you will need to get started.

Please let me know if you have any questions.

Thanks. Posted by Peskoe Ben(unfuddle username: bpeskoe)

[Battler45]

George,

In implementing new updater system for region-25, you might want to refer to the existing updater system implementation (especially for field mappings). I've attached a zip file containing the source-code for the same. It contains the following folders:

mibor - contains code specific to updater-system for region 25. The "mls_autoupdates.asp" file contains the entry-point; that is, the programming in this file will initiate the updates for region-25. It will call appropriate functions like UpdateListings, RemoveOldListings, UpdatePhotos etc... for performing various update steps. Some of these functions could be defined in the mls_functions.asp file present under this folder. And, rest of the functions would be defined in the common code.

res/includes - contains code common to all updater-systems. The mls_functions.asp file for example defines InsertUpdateMLSListing function, which is called to add/update listing records.

---

I'm not sure if you had a chance to work with libRETS library. If you haven't, I would recommend first creating a sample project that uses the libRETS library. Pls use the above RETS credentials for connecting to the MIBOR RETS server. Ideally, you should first download the metadata from this server. That would help you in mapping listing fields.

Pls note that the MIBOR region data is stored in the sierrainteractive-1 database. You can copy the latest back up of the database (development) from the following FTP location:

Host: 75.103.116.57 User-name: george_dev_db Password: GeorgeM$123

Since the development DB is has restricted access, pls restore the above back-up on your local DB and configure the updater code to work with your local DB.

Pls let me know if you have any queries.

Thanks.

Posted by Shah Vihar(unfuddle username: vshah)

[Battler45]

George,

To further clarify, the existing updater for mibor code will probably be confusing to you. It references the sys IDs of each database field, rather than the long field names, which is what you will be using when you query the RETS server using libRETS (I believe). You can use the T5 Field Info RETS Updated spreadsheet attached to the first comment in this thread to help you see which sys IDs match to which long field names.

In bringing any new data feed in line with the new updater system, the most difficult part will always be to determine how the fields from the host RETS database should be mapped into the fields of our own database.

In certain cases, such as when dealing with City, State, and Zip Code, these are straightforward.

However, in dealing with other types of listing data, such as interior and exterior features, lot features, etc., this can be more complicated. So, in working this out, referring to the existing SoCal MLS updater mappings (region 10) may also be quite helpful for you, because that region will (I believe) contain many of the same field names and will share many similar mappings.

Please let me know if you have any questions.

Thanks. Posted by Peskoe Ben(unfuddle username: bpeskoe)

[Battler45]

George,

Yes, the specified RETS servers are live. I think it should be ok to connect to those servers for testing purpose.

I understand that you want to test the new updater system - the SoCal updater system to be specific. To do so, you can run the WindowsServiceTest console application. The UpdateProcessor.cs file in this project contains the entry-point function, PerformUpdatesForRegion. This function will execute appropriate updater steps (depending on the steps supported for a region - as defined in the Updater_MLSRegionUpdateSteps table). If you want to test a specific step, pls comment out appropriate code in the PerformUpdatesForRegion function.

Also, you would need to change certain configuration parameters defined in the app.config file. Some of these parameters are:

• Connection-Strings: updaterConnString & connString1 - pls set these to point to your local db server.
• PhotoRootFolder: this is the path where photos would be downloaded from the MLS server
• LogEmails: true / false. If set to true, alert-emails would not be sent to actual email addresses; they would instead be logged in the log file
• TestEmailAddress: if the LogEmails parameter is set to true and this parameter's value is set then emails would be sent to this test email address as well.

Pls let me know if you need additional info.

Thanks.

Posted by Shah Vihar(unfuddle username: vshah)

[Battler45]

Hi Vihar, Could you please explain more detailed how I can test the application? As I understand the specified RETS servers are live not "sandbox"? Is it ok to connect to servers for testing purposes? Thanks Posted by Molchanov George(unfuddle username: george)

[Battler45]

George,

Yes, it's definitely okay to connect to live servers for testing purposes. Since the data feed only comes in one direction (from them to us), it's fine to query during the testing process.

Thanks Posted by Peskoe Ben(unfuddle username: bpeskoe)

[Battler45]

George,

I see -- to give you a bit of background, there are a number of different data providers around the United States. These are companies that build and maintain the software for individual boards of realtors. The Socal MLS and the Indianapolis MLS (MIBOR) both use a company called MarketLinx.

This means that, although some of the fields and field names may vary between SoCal MLS and MIBOR MLS, most of them will be the same, and the way you query the database for MIBOR will be very similar to the way the database is queried for the existing SoCal MLS updater, and so you should feel free to look at SoCal MLS updater as a reference for MIBOR. (Other boards of realty use different data providers, and so this will not always be the case.)

That said, I am not sure of the answer to your question, but I believe Vihar should be able to answer this for you, since I am sure it is the same in MIBOR as it is in SoCal MLS. (My guess is that "Change Date" is the correct answer, but I am not certain.)

Ben Posted by Peskoe Ben(unfuddle username: bpeskoe)

[Battler45]

Vihang,

Since we need to test content for the sites in development and we'll be entering that content in the live DB, could you restore the development sites to pull content from the live DB rather than the development DB?

Thanks. Posted by Peskoe Ben(unfuddle username: bpeskoe)

[Battler45]

George,

Just checking in to see how things are coming -- were my last comments helpful, and have you been able to make further progress?

Thanks. Posted by Peskoe Ben(unfuddle username: bpeskoe)

[Battler45]

Attachment: Posted by Shah Vihar(unfuddle username: vshah)

[Battler45]

Ben, Vihar,

I have some questions regarding outstanding issues (geocodes and open houses details).

1. Geocodes. It looks like old updater doesn't get latitude and longtitude from MIBOR response. Isn't? I found in meta data some fields which probably can be used for this - DrivingLatitude and DrivingLongitude but I'm not sure. Anyway in my tests all MIBOR records don't return values for these fields. Also I've added functionality of geocode processing (similar to SoCal functionality) and it works.

2. Open-house updating. Can it be that MIBOR's user doesn't have some rights to get open house data? I've tried to get such data and I received error "User does not have access to Resource named OpenHouse".

Thanks Posted by Molchanov George(unfuddle username: george)

[Battler45]

Vihar,

Unfortunately I can not remote into 208.106.250.174 to update database with my scripts - maximum number of allowed connection is exceeded. Also I have again question about git - I've run git fetch command but I don't see any changes (there should be changed ImageHelper.cs and Constants.cs). What I have to do more?

Thanks Posted by Molchanov George(unfuddle username: george)

[Battler45]

Vihar,

RE: #63 - Your alternate approach does sound good and workable. I guess that appending a querystring parameter to the photo filename does not impact the ability of the browser to render the images? (I've tested, and it does not seem to be a problem.) In that case, this does seem like the best and easiest solution.

I have not yet checked when and how the DateAdded field in the MLSPhotos table is updated, but I will do so tomorrow, and will make sure that the old updater is set to update this field appropriately. Can you check the new updater to make sure it is also updating this value appropriately?

Thanks Posted by Peskoe Ben(unfuddle username: bpeskoe)

[Battler45]

Vihang,

I've already done this -- updated the dev sites on DEDN173 to pull content from the live db rather than the development db (I need to work on a new site today).

Thanks Posted by Peskoe Ben(unfuddle username: bpeskoe)

[Battler45]

Status of task: I'm able to get response from RETS server and going to parse it.

Ben, I have question about query for request: which date should be used in query to select only outstanding records? For example table "RES" has several date fields - "List Date", "Entry Date" and "Change Date". Do you know which is correct? Posted by Molchanov George(unfuddle username: george)

[Battler45]

Vihar, I got RETS meta-data with using of libRETS code samples. Attached are archive with meta-data in text and xml format. In my opinion spreadsheet T5field_info_RETS_Updated01-17-09.xls looks more complete. For example ResidentialProperty object: Field #3: in spreadsheet it has name "Acres", in meta-data isn't any standard name for this field ([] - it means standard name is missed). I thought I can use this value to fill field Acreage of Property object. Is my assumption wrong? Posted by Molchanov George(unfuddle username: george)

[Battler45]

George,

Sorry for the delayed response -- yes, you are correct that for the initial data load, we will need to import all photos new from the server (this process will likely take a couple of days).

I have been talking with Vihar for the past couple of days, and we are going to change the max dimensions of the large size of the imported photos. Up to now, we have allowed a maximum width or height of 550px, and going forward we are going to allow a max width or height of 640px. Vihar may have already made this change in the code, but please confirm with him before you begin the photo download.

Thanks.

Ben Posted by Peskoe Ben(unfuddle username: bpeskoe)

[Battler45]

George,

I've logged off from the server. Can you pls try after 5 mins and let me know if you can log into the server?

As for the changes to be applied to your local copy - you'll need to merge the remote changes into your local branch. This can be done by executing the following command git merge origin master. I'm not sure if origin is the name of your remote branch. You can check the same by executing git branch -r command; this will list all remote branch maintained by the git repository.

Alternatively, you can run git pull origin master. This will first fetch remote changes and immediately merge them in your local repository. So in a way it does the job of git fetch and git merge.

Pls let me know if you have any queries.

Thanks Posted by Shah Vihar(unfuddle username: vshah)

[Battler45]

Ben, It is very helpful! Thank you!

Posted by Molchanov George(unfuddle username: george)

[Battler45]

Ben,

Developing a small application for data comparison sounds good. The only concern I've is that in using this application, the old and new updater would have to be synchronized. That is, the tests would have to be carried out only after the old updater has run; otherwise the comparison application won't be much of a use. Could you pls confirm the time when the MIBOR updates are completed (through the old updater)? George can then schedule the new updaters to run around that time, and, then can use comparison application to verify correctness of data. Pls let me know your thoughts on this.

Thanks. Posted by Shah Vihar(unfuddle username: vshah)

[Battler45]

George,

We needed to change the data-types of Latitude / Longitude columns (Geocodes table) to numeric(12,9). This led to change in a couple of updater specific store-procedures, and, hence, change in the new updater system. I've made these changes, and, have pushed the same to the updater repository. Pls fetch the latest changes and update your local copy.

Pls note that you'll also need to make a minor change in the way latitude / longitude values are set in the MIBOR import. Until now these values were set as string, but, now they would need to be set as Double. Could you pls update MIBOR import code to set latitude / longitude values as double. You can use the ToDouble string extension I've used in SoCal importer. Pls let me know if you have any queries.

Pls note that I've applied these changes to the dev. sierrainteractive-1 DB. I've attached an SQL script that you can run on your local DB to apply data-type specific changes.

Thanks. Posted by Shah Vihar(unfuddle username: vshah)

[Battler45]

Ben, Vihar,

I completed test of MIBOR updater on development server. Update process was executed without errors. I don't understand how to test results using of comparison of live (www.hometoindy.com) and development (hometoindy.sierrainteractivedev.com) sites. I see that some record is added into database - I can find this record by its MLS number in database sierrainteractive-1. But when I'm trying to find this records on sites (with using of search by MLS number) I could not get any results. Does development site use database sierrainteractive-1 on sql4.sierrainteractivemls.com indeed? Anyway I'm not sure that visual checking will give an opportunity to find out all issues (if these exist, I hope new updater is working without issues). Perhaps it will be more useful to make a small application intended to compare databases?

Thanks Posted by Molchanov George(unfuddle username: george)

[Battler45]

George,

You can find the example for working with RETS meta-data here: http://www.crt.realtors.org/projects/rets/librets/documentation/devguide/. The 3rd point, "Working with metadata" provides some examples of downloading RETS server metadata. This metadata can then be used as reference for mapping MIBOR fields to our database fields.

Thanks. Posted by Shah Vihar(unfuddle username: vshah)

[Battler45]

Ben,

The only other open point is re. import of Open House details. As George had mentioned earlier, it looks like our MIBOR RETS account doesn't have access to open-house details. Pls let me know if the open-house step can be ignored for MIBOR region.

Thanks. Posted by Shah Vihar(unfuddle username: vshah)

[Battler45]

Ben, Vihar,

I've completed latest minor changes (Driving Longitude / Driving Latitude, ListTypeDescrip).

Also I've added IndianapolisListingsPurger class to delete records but there is again known issue - User does not have access to Resource named DeletedListing.... Resource name is correct (according meta data), but it doesn't work. And we have not other ways to delete records as we getting only active - we can not get records with other statuses.

What we will do?

Thanks Posted by Molchanov George(unfuddle username: george)

[Battler45]

George,

We needed to change the data-types of Latitude / Longitude columns (Geocodes table) to numeric(12,9). This led to change in a couple of updater specific store-procedures, and, hence, change in the new updater system. I've made these changes, and, have pushed the same to the updater repository. Pls fetch the latest changes and update your local copy.

Pls note that you'll also need to make a minor change in the way latitude / longitude values are set in the MIBOR import. Until now these values were set as string, but, now they would need to be set as Double. Could you pls update MIBOR import code to set latitude / longitude values as double. You can use the ToDouble string extension I've used in SoCal importer. Pls let me know if you have any queries.

Pls note that I've applied these changes to the dev. sierrainteractive-1 DB. I've attached an SQL script that you can run on your local DB to apply data-type specific changes.

Thanks. Posted by Shah Vihar(unfuddle username: vshah)

[Battler45]

Vihar,

Would you mind taking a look at the data on the development server, including the photos and geocodes, and testing to make sure it matches the live region 25 data?

If so, then I suppose we don't need to re-download all photos and geocodes for all Indy listings -- could we just use the new updater system to pick up from where the old updater system leaves off -- perhaps we could schedule that for Thursday morning this week if everything looks good?

Thanks

Posted by Peskoe Ben(unfuddle username: bpeskoe)

[Battler45]

George / Vihar,

RE: #53

You are correct -- Open Houses are not available for MIBOR and so can be excluded from the update process.

Thanks Posted by Peskoe Ben(unfuddle username: bpeskoe)

[Battler45]

George,

I've made changes to the image-processing code to generate large images in 640x640 dimensions. I've pushed this change to the master branch. Can you pls fetch the latest code and update your local code?

This just for your info - I've created a new branch, production in the updater git repository. This branch will contain code that is currently used for production updater service. That is, it won't contain any development specific code. I'll maintain this branch to make sure that development specific changes are not pushed to the production service. Pls continue pushing your changes to the master branch (which would contain all development specific changes like you are making for implementing MIBOR updater).

As for photo-download/process testing - I would suggest you to first test it on your local machine (if you haven't already). For this, pls set the LastUpdateTimeStamp for photo-step (16) to current date. This will make sure that only most recent photos are downloaded/processed. Once you've verified the changes, pls test the MIBOR updater on the development server. If everything is working as expected, we can then download all records / photos (by setting the LastUpdateTimeStamp value to null for all steps). Pls let me know your thoughts.

Thanks.

Posted by Shah Vihar(unfuddle username: vshah)

[Battler45]

George,

A few points for your info:

1 -- The SoCal and Indianapolis Geocoder classes were same so I've removed one of them, and, have renamed other as DefaultGeocoder class. It will be used in Geocoding listings for both these regions. In future, if any region's Geocoding step diverges, a new class can be implemented for that region; otherwise we can continue to use the DefaultGeocoder class.

2 -- I noticed that the DateListed value (which represents the data when a property was listed, and is used in calculating DaysOnMarket value) was always set to current date. Apparently, the format of the ListDate field in the RETS was different. I've made changes accordingly. It now correctly sets DateListed and DaysOnMarket values.

3 -- For each region, a couple of addition steps need to be executed viz. Statistics Calculations (it calculates stats like average price / median price etc...) and Generating Email Alerts (sending emails to leads who have opted to be notified when new listings matching their criteria are added). These steps are common; no additional programming is required. We'll just need to associate the region with the corresponding step in the Updater_MLSRegionUpdateSteps table. The step-ids are 21 and 20 respectively. I think the email-alert step (20) can be associated once we have thoroughly tested the MIBOR region updater.

4 -- To make testing of a single region easier, I've added a configurable parameter, MLSRegionToTest in the console-application. When its value is set to one of the region-ids (i.e. 25) it will perform updates for that specific region. When set to -1, it will perform updates for all regions. I think this would be useful as when we convert regions to new updater system.

Thanks.

Posted by Shah Vihar(unfuddle username: vshah)

[Battler45]

George,

Sorry for the delay in reply. I agree that Standard name is missing in several fields. I guess we could make use of other descriptive fields like short-name / description to get info. about fields. I've downloaded RETS metadata, which includes such description values. Pls find attached the file listing the entire MIBOR RETS metadata. Pls note that we would mainly be interested in "Property" resources (line# 4092). Each "Class Name" under the "Property" resource would represent type of listing e.g. Residential Lease, Residential/Condo, Commercial Retail, Commercial Land and so on...

The pattern for class-name is as follows: Class name: class-description (class-system-name) [class-standard-name]. For example, Class name: Residential Lease(1) [], Class name: Vacant Lots/Land(4) [LotsAndLand]

The pattern for table-name is as follows: Table name: table-short-name (table-system-name) [table-standard-name]. For example, Table name: Base Desc(21) [], Tot FB(24) [BathsFull].

You can use this information for mapping sierrainteractive db fields to corresponding RETS fields. In addition, you can use the system-name / standard-name for querying RETS data-feed.

As for finding recently updated records, pls use the ChgDt (151) (line# 4690) value to query RETS server.

Pls let me or Ben know if you have additional queries in mapping other fields (especially composite fields)

Thanks.

Posted by Shah Vihar(unfuddle username: vshah)

[Battler45]

Vihar / George:

RE: #52

Unfortunately, the performance of the existing photos server (DEDQ61) is not at an acceptable level, and so we will need to move all of those photos, either to the new cloud server (where the SoCal photos are) or to a 3rd party service like Amazon or Azure.

The DEDQ61 server is a very old server with an under-powered hard drive, and so at peak times the photos are very slow to load. My goal is that, once we are able to covnert all existing regions over to the new updater system, we will be able to retire the DEDQ61 server.

I do agree that the best long-term approach would be to host the photos on a 3rd-party service, either Amazon or Azure. Do you have any preference toward one or the other. I am really not sure of the technical details of using such a service in combination with our service, and I'm also not sure how to determine the cost. Is that something you could help me understand?

Perhaps we could use MIBOR as a test case for moving photos over to one of these services, with the goal of eventually moving all photos over.

The one issue I have had thus far which has made it difficult to use a CDN is photo expiration. Currently, each listing has a set of photos which gets distributed out via a CDN powered by Level 3. However, because agents change photos for listings on a regular basis, and these updated photos often have the same filename as the original, I have not found a good way to "expire" the old photos when they have been updated, other than to set the expire (TTL) value to 1 day for all photos.

This means that all photos are re-loaded into the CDN once per day no matter what, and creates a much heavier load on the server than there needs to be.

Do you have any thoughts on how this might be improved? My only idea is that when new (replacement) photos are received for an existing listing, they should always be given a new filename. This would eliminate the above problem and ensure that the newer photos are always displayed for every listing, and it would also allow us to cache all photos on the CDN for a much longer time (30 days).

If we do implement something like the above, we just need to make sure that the proper order for the photos is preserved using the OrderID field in the MLSPhotos table.

I agree that this is a pressing issue and so if you could continue to split your time between this topic and the clustering issues, that would be great.

Thanks. Posted by Peskoe Ben(unfuddle username: bpeskoe)

[Battler45]

Vihar,

Thank you. I able to connect to 208.106.250.174 now.

Thanks Posted by Molchanov George(unfuddle username: george)

[Battler45]

Vihar, George,

RE: #54

1 - Yes, please do not use Driving Longitude / Driving Latitude values from the MIBOR feed. Instead, use our Geocoding scripts which will geocode listings directly based on property address.

2 - For ListType 1 (Residential) the field is 511: Res/Cnd for ListTypeDescrip. For ListType 4 (Land) the field is 343: Type for ListTypeDescrip. For ListType 5 (Multi-Faimily), the field is 135: List Type for ListTypeDescrip.

3 - Vihar, I believe you are correct and that our access is restricted to only be able to retrieve Active / Pending listings. However, it would still be nice to be able to track listing status if we can using the Status field in the MLS table. (In our front-end application, we may at some point want to display status to end-users.)

4 - Regarding usage of

, that was an attempt on my part to improve the way the listings are displayed on the front end. However, it was a failed experiment, and so it's fine to format the Features* values in the same way that they are formatted in the SoCal MLS data feed. In fact, it would be preferable to keep it the same as the SoCal format for consistency's sake.

5 - I'm not sure about this. In some cases, an agent my replace existing photos with a new set of photos, but the overall number of photos may not have changed. In such cases, it would definitely be necessary to delete all existing photos and re-download the full set to insure that all new photos are properly received.

Please let me know if you have any questions

Thanks Posted by Peskoe Ben(unfuddle username: bpeskoe)

[Battler45]

George,

You are correct. The dev. site also connects to the live database. That is the reason it didn't display any result for MLS search. Ben and Vihang are working on adding new front-end sites, and, therefore, they need to connect the dev. sites (on sierrainteractive-1) to live DB so we would not be able to perform visual comparisons.

Btw, I quickly checked the latest MLS records on the dev DB. Pls find below my comments:

1 -- It looks like the newly added listings are not geocoded. That is, the Geocode_ID value for such listings is 0. To give you a brief overview of geo-coding - we have a table, Geocodes, which stores details like latitude, longitude, address etc... This info. is used in displaying listing on map, performing search by map area etc... So when a new listing is added to the MLS table, we also associate it with a new Geocodes record. If latitude / longitude info. is available from the RETS server, then those values are used in creating a new Geocodes records.

The updater also runs Geocoding step. This step queries 3rd-party service like Yahoo / Google to get missing geocoding details. And, associates listings with new Geocodes records. So for all those listings, which don't have latitude / longitude info. available, the geocoding step populates missing details for them.

Can you pls verify this in the MIBOR updater?

2 -- None of the listings have open-house details. Can you pls check if the open-house step is executed for the MIBOR region? This step should populate info. like open-house date / time etc...

3 -- I'm not sure if had the photo-downloader step enabled for this specific test. If it was, can you pls check why photos have not been downloaded for newly added listings? Also, no records have been added in the MLSPhotos table.

4 -- When you have chance, can you pls add some logging in the MIBOR steps' programming? This will help us in verifying updater steps, location errors (if any).

I'm currently working on finishing one of the open tasks. Hopefully that would be completed in another day or two. I'll then be able to test the updater thoroughly.

Thanks.

Posted by Shah Vihar(unfuddle username: vshah)

[Battler45]

Ben, Vihar I added code to photos downloading. As I understand ASP system we have not an opportunity to update photos (like in SoCal importer) - we have to delete all photos for property and download again. Is it correct? Thanks P.S. I will work with remote testing tomorrow Posted by Molchanov George(unfuddle username: george)

[Battler45]

Ben,

As for caching photos on CDN for longer time - renaming photos to new name might affect some of the existing functionality (at least the new updater). A check is performed to update photo records based on the photo-name value. That said, I think it would be straightforward to update this piece of programming. Pls let me know if similar checks are performed in the old updater (and front-end sites).

Another point to consider would be photo reordering step - I understand that this step is executed for region-3 and so it may need to be tweaked to handle the change in photo-names.

An alternative approach would be to append timestamp to photo-url. For example, http://photos2.sierrainteractivemls.com/10/mlspics/8C/10_U10002739_05.jpg?ts=20110908. The MLSPhotos table does have a date field, DateAdded; its value can be used as timestamp. I'm not sure where and how this field is used but if it can be updated when a photo is re-downloaded, we can use its value in generating photo-urls. I understand that the spUpdateMLSPhoto is executed when an MLSPhotos record is to be updated. This sp can be modified to set update-date in the DateAdded field. And, the photo-url generation code will append new value thus invalidating the cache. Once this is done, CDN settings can be changed to cache images for a longer period (even a year).

Pls let me know your thoughts.

Thanks.

Posted by Shah Vihar(unfuddle username: vshah)

[Battler45]

Ben,

Yes, in moving to new updater for MIBOR region, it would need to pick up from where the old updater leaves off. I believe except for the photo processing / storing, rest of the processing would be pretty much unaffected. That is, the new updater system will not delete any existing listings records so the leads' associations with the current listings will remain unchanged.

As for photo processing, I believe it would be good if we can make use of the existing photos; thus eliminating the need to migrate (re-download) photos for existing listings. One possible approach would be to introduce a new field in the MLSPhotos table say, PhotoServer (tinyint). This field will indicate the server where the photo would be stored. The existing code that builds path for listing images already handles two different paths (i.e. for SoCal regions and for the rest of the regions) so we would just need to change that programming to take into account the PhotoServer value.

Per this approach, the PhotoServer value for existing photo-records would be say 1 for region-10 records and 0 for the rest of the records. Any new photos processed through the new updater system will have the PhotoServer value set to 1. So the front-end site programming will use the PhotoServer value to build listing photo url. It currently makes distinction between SoCal region and other regions, which can be changed to make distinction based on the PhotoServer value.

The only issue with the above approach would be to handle removal of photos on old server through the new updater system. When an existing listing record is removed, its associated photos are also removed. When the "Purging" step is executed for MIBOR region (in the new updater system), it may be possible that all or some of the photos of the listings to be purged are present on the old update server. My thought is that this would need to be handled through FTP interface.

We can go one step further and implement programming, which would be flexible enough to handle additional photo servers. I haven't given this much thought but I realize that we would have to think about this sooner as the SoCal images are already occupying about 30GB of hard-disk space on the Crystaltech cloud server, and MIBOR images will only add to this size. I guess it would be possible to increase the size so we can continue with the above mentioned approach.

I also wanted to discuss possibility about hosting photos on 3rd-party services e.g. combination Amazon S3 (for hosting photos) & Amazon CloudFront (for CDN) / Azure Storage & Azure CDN. If this is viable then we could change the new updater system to store photos on such service. This will eliminate the need to ramp-up resources on updater servers. And, in the long run (after we have moved to new updater system completely) all photos would be hosted on such CDN service, which would definitely reduce traffic / load on the servers.

Pls let me know your thoughts. George - pls let me know if you have any suggestions / recommendations for the same.

Thanks.

Posted by Shah Vihar(unfuddle username: vshah)

[Battler45]

Vihar,

I'm working now on the photo-migration tool. I hope the first revision will be ready today.

Thanks Posted by Molchanov George(unfuddle username: george)

[Battler45]

Vihar, Thanks for your help. I will check this listing. Regarding composite fields for Residental Property. Now I'm able to fill next composite fields with same headers as for SoCal:

• FeaturesEnergy
• FeaturesInterior (only data for "Eating Areas" and "Interior")
• FeaturesLot (only data for "Lot Description" and "Lot Size Dimension Description")
• FeaturesFinancial
• FeaturesFoundation

For other composite fields (FeaturesCommunity, FeaturesCondo, FeaturesConstruction, FeaturesDocuments, FeaturesExterior, FeaturesFencing, FeaturesHOA, FeaturesParking, FeaturesRoof, FeaturesUtilities) I can not find MIBOR fields which could be assigned to SoCal headers. Probably some composite fields can be filled with MIBOR data with other headers (for example MIBOR contains "Garage description" field which can be placed into composite field FeaturesParking). But I'm not sure is it legal or not and who should make such decisions. Posted by Molchanov George(unfuddle username: george)

[Battler45]

Ben,

I believe George hasn't yet run the MIBOR updater on the dev DB so it should be fine. I'll confirm with George and restore the back up based on his response.

George - can you pls confirm if it would be alright to go ahead with the restore of sierrainteractive-1 DB?

Thanks.

Posted by Shah Vihar(unfuddle username: vshah)

[Battler45]

Vihar,

Can you explain to me how you are able to delete listings from SoCal? Since the same company provides the data feed service for both SoCal and MIBOR, hopefully we can use the same method. If we do not have access to a needed resource, let me know and I will contact the MIBOR representative and Marketlinx to see if we can get access as needed.

Thanks Posted by Peskoe Ben(unfuddle username: bpeskoe)

[Battler45]

George,

Sorry for the delay getting back to you on this -- I am working on it now and will have it for you by the end of the evening (my time). In the meantime, if you have more time to work on the project, perhaps you could begin looking into the photo downloads for this region? That (hopefully) should be very similar to the way it is currently being done for SoCal MLS / Region 10.

Thanks. Posted by Peskoe Ben(unfuddle username: bpeskoe)

[Battler45]

Vihar,

Re: #61 - From what I can tell, pricing is fairly similar. Unfortunately, I don't have access to the type of data we would need to accurately assess the monthly costs, but even if it turned out to be double your estimate, that would be do-able for us and worth it if it allows us to provide customers with images that always serve quickly, and reduces the load on our dedicated servers (and thus reduces the need for additional servers).

I'm running a file tree right now which will tell us exactly the total size of the MLS folder on DEDQ61 -- I believe it is 50-60 GB currently, so I think we are still under 100GB total for all photos.

By the way, 30GB does seem like a lot for the SoCal photos -- the overall size of the storage on the server for this region should remain fairly consistent over time, shouldn't it (aside from the increase in the size of the large photos that we introduced recently)? Can you double-check to make sure that when a listing is deleted from the server, all associated photos are also getting deleted?

(Also, can you please restart DEDN173 overnight tonight -- it needs to be restarted following updates.)

Thanks Posted by Peskoe Ben(unfuddle username: bpeskoe)

[Battler45]

Ben,

RE: #65: Removal of listings in SoCal is performed in two-steps:

1 -- soft-removals - whenever a listing's status is changed, that change is recorded in the "History" resource. The SoCal removal step queries for all listings whose status has been changed to one of the inactive statuses (i.e. Withdrawn, Expired, Closed etc...).

2 -- hard-removals - in certain cases, listing records are physically removed from the RETS database. Details of such listings are stored in the "Deletions" resource. The SoCal removal step queries for all listings, which might have been removed from the RETS server.

Any listing found through one of the above steps is then removed from the sierrainteractive database (along with its associated records).

I verified that the MIBOR RETS meta-data does have corresponding History (PAR) and DeletedListing resources but we don't have access to these resources. Looks like our user-account may have very limited access in that several resources like OpenHouse, Media, History, DeletedListing are not accessible. Can you pls confirm with MIBOR representative about the same?

Thanks.

Posted by Shah Vihar(unfuddle username: vshah)

[Battler45]

Ben, Vihar,

I tested photos downloading ang calculation of geocodes on development server. It works.

Thanks Posted by Molchanov George(unfuddle username: george)

[Battler45]

George,

1 - Yes, it's fine to geocode listings using our own functionality -- that's what I am also doing in the current updater.

2 - Yes, I agree, I believe that is the case. I will check tomorrow with MIBOR staff to confirm, but for now, please proceed as if that data will not be available.

Thanks. Posted by Peskoe Ben(unfuddle username: bpeskoe)

[Battler45]

George & Vihar,

I do think it might be worthwhile to develop a small application which would compare data rows between dev and live DBs for the same listing to make sure that the data is the same -- mainly because we do have a number of additional regions to convert from the old updater to the new updater, and so this application would be used for all of those regions and would therefore save us quite a bit of time in testing.

Unless, Vihar, you have already written some scripts that will do this type of comparison or have something else in mind?

Thanks. Posted by Peskoe Ben(unfuddle username: bpeskoe)

[Battler45]

RETS meta-data Posted by Molchanov George(unfuddle username: george)