Convert Region 25 (MIBOR / Indy) to new updater

[Battler45]

George,

The first project on which you'll be working will be to convert an existing MLS data feed to the new updater system.

Vihar will be able to provide you with the technical details for installing GitHub and downloading a local copy of the current code base.

Vihar, will you also please provide George with whatever server and / or database access he needs for this initial project? Please share with him any necessary logins and create any new users as necessary. Also, I think that converting region 25 will be a good place to start because it shares the same MLS data provider (Marketlinx) as the one you have already converted (SoCal MLS). You can find my existing code in:

C:\Inetpub\mls4.sierrainteractivemls.com\admin\mibor

on DEDQ61

The login credentials to the RETS server are:

• UserID: 31041rets

• Password: BENJ1001

• UserAgent: sierrainteract

• Password: h1ghplateau

RETS URL: http://inr.rets.interealty.com/Login.asmx/Login

As you'll see when you examine my existing code, I was using the sysid name for the database fields rather than the long names for the fields (which were being used for SoCal MLS). I did this because I had a lot of difficulty with the RETS Connector software that I was using to download the daily data files. To help translate between the two, I've attached a spreadsheet provided to me by MIBOR (which stands for Metropolitan Indianapolis Board of Realtors) that cross-references the long field names with their numeric IDs.

Please let me know if you have any questions.

Thanks.

Posted by Peskoe Ben(unfuddle username: bpeskoe)

[Battler45]

George,

Yes, the specified RETS servers are live. I think it should be ok to connect to those servers for testing purpose.

I understand that you want to test the new updater system - the SoCal updater system to be specific. To do so, you can run the WindowsServiceTest console application. The UpdateProcessor.cs file in this project contains the entry-point function, PerformUpdatesForRegion. This function will execute appropriate updater steps (depending on the steps supported for a region - as defined in the Updater_MLSRegionUpdateSteps table). If you want to test a specific step, pls comment out appropriate code in the PerformUpdatesForRegion function.

Also, you would need to change certain configuration parameters defined in the app.config file. Some of these parameters are:

• Connection-Strings: updaterConnString & connString1 - pls set these to point to your local db server.
• PhotoRootFolder: this is the path where photos would be downloaded from the MLS server
• LogEmails: true / false. If set to true, alert-emails would not be sent to actual email addresses; they would instead be logged in the log file
• TestEmailAddress: if the LogEmails parameter is set to true and this parameter's value is set then emails would be sent to this test email address as well.

Pls let me know if you need additional info.

Thanks.

Posted by Shah Vihar(unfuddle username: vshah)

[Battler45]

Status of task: I'm able to get response from RETS server and going to parse it.

Ben, I have question about query for request: which date should be used in query to select only outstanding records? For example table "RES" has several date fields - "List Date", "Entry Date" and "Change Date". Do you know which is correct? Posted by Molchanov George(unfuddle username: george)

[Battler45]

George,

I see -- to give you a bit of background, there are a number of different data providers around the United States. These are companies that build and maintain the software for individual boards of realtors. The Socal MLS and the Indianapolis MLS (MIBOR) both use a company called MarketLinx.

This means that, although some of the fields and field names may vary between SoCal MLS and MIBOR MLS, most of them will be the same, and the way you query the database for MIBOR will be very similar to the way the database is queried for the existing SoCal MLS updater, and so you should feel free to look at SoCal MLS updater as a reference for MIBOR. (Other boards of realty use different data providers, and so this will not always be the case.)

That said, I am not sure of the answer to your question, but I believe Vihar should be able to answer this for you, since I am sure it is the same in MIBOR as it is in SoCal MLS. (My guess is that "Change Date" is the correct answer, but I am not certain.)

Ben Posted by Peskoe Ben(unfuddle username: bpeskoe)

[Battler45]

Hi Vihar, Could you please explain more detailed how I can test the application? As I understand the specified RETS servers are live not "sandbox"? Is it ok to connect to servers for testing purposes? Thanks Posted by Molchanov George(unfuddle username: george)

[Battler45]

Ben, Vihar I have some questions regarding common rules of fields' mapping. 1) What should I do if I can not find an appropriate source field for some field of object Property (for example SoCal model has data for ListAgentLastName - p.AgentLastName = resultSet.GetString("ListAgentLastName"), - but I can not find such field in MIBOR's fields). 2) On the other hand there are a lot of MIBOR fields which are not mapped with object Property. 3) Special case is the composite fields (like FeaturesExterior). Such fields in SoCal implementation have fixed headers for specific source fields (for example in composite field FeaturesExterior can be added field PatioFeatures with header Patio). Are these headers fixed or for MIBOR field FeaturesExterior can contain other data with other headers? And if yes - do you have some ideas which MIBOR fields can be included in composite fields of object Property? Posted by Molchanov George(unfuddle username: george)

[Battler45]

George,

I agree - it should be "Change Date". If we filter records by that date, the server should return records that have been changed since the specified date. Also, can you pls share the MIBOR meta-data? I'll use it as reference for calrifying / confirming other this and mappings.

Thanks. Posted by Shah Vihar(unfuddle username: vshah)

[Battler45]

George,

I've set you up so that you should be receiving updates on this task / thread -- please confirm. This thread contains the initial details you will need to get started on your first task, and Vihar can provide you with the background, technical details, etc. that you will need to get started.

Please let me know if you have any questions.

Thanks. Posted by Peskoe Ben(unfuddle username: bpeskoe)

[Battler45]

Vihar, I got RETS meta-data with using of libRETS code samples. Attached are archive with meta-data in text and xml format. In my opinion spreadsheet T5field_info_RETS_Updated01-17-09.xls looks more complete. For example ResidentialProperty object: Field #3: in spreadsheet it has name "Acres", in meta-data isn't any standard name for this field ([] - it means standard name is missed). I thought I can use this value to fill field Acreage of Property object. Is my assumption wrong? Posted by Molchanov George(unfuddle username: george)

[Battler45]

George,

You can find the example for working with RETS meta-data here: http://www.crt.realtors.org/projects/rets/librets/documentation/devguide/. The 3rd point, "Working with metadata" provides some examples of downloading RETS server metadata. This metadata can then be used as reference for mapping MIBOR fields to our database fields.

Thanks. Posted by Shah Vihar(unfuddle username: vshah)

[Battler45]

George,

Just checking in to see how things are coming -- were my last comments helpful, and have you been able to make further progress?

Thanks. Posted by Peskoe Ben(unfuddle username: bpeskoe)

[Battler45]

George & Vihar,

We received an email today from a staff member at MIBOR / Indy stating that we are displaying listings on the HomeToIndy.com site which should not be displayed.

In the data feed, there is a field, "Public Internet Yes/No". When this field is set to "No", we are not meant to include these listings in our feed and display them in the site.

The staff member cited the listing MLS #21163069 as one such listing for which this field is set to No, but which is displayed on the HomeToIndy.com site.

Can you please check the programming for the Indy Updater and correct this issue, and then re-run a full update of all Indy listings so that these "No" listings are no longer displayed on the site.

This is a priority for us, and it would be good if we could have this done by tomorrow (Wednesday), my time.

Thanks. Posted by Peskoe Ben(unfuddle username: bpeskoe)

[Battler45]

Ben,

I believe George hasn't yet run the MIBOR updater on the dev DB so it should be fine. I'll confirm with George and restore the back up based on his response.

George - can you pls confirm if it would be alright to go ahead with the restore of sierrainteractive-1 DB?

Thanks.

Posted by Shah Vihar(unfuddle username: vshah)

[Battler45]

Ben & Geroge,

I've tested the updated listing-removal functionality. It is working fine. It pretty quickly downloads 28K listing ids (within a few seconds) so time should not be a concern. That said, I realize that the existing MLS records for MIBOR region don't have valid SourceUniqueID values, and, therefore, removal operation will fail for them. I thought about getting MLS-numbers to address this issue but apparently the request failed with some xml-parsing error. I'd read that the RETS operation involving large record-set works fast only when unique-keys are to be returned so that might be the reason for the failure as MLS-numbers are not unique.

An alternative would be to change the listing-removal functionality to retrieve MLS-numbers in batches. George - can you pls look into updating the listing-removal functionality to retrieve MLS-numbers of all active listings? You will most probably need to use offset in this case.

Ben - another work-around would be to populate SourceUniqueID value for all existing MLS MIBOR records through a one-time operation. Once we have unique-ids populated, we can go ahead with the existing implementation.

Pls let me know your thoughts.

Thanks.

Posted by Shah Vihar(unfuddle username: vshah)

[Battler45]

George,

Yes, it's definitely okay to connect to live servers for testing purposes. Since the data feed only comes in one direction (from them to us), it's fine to query during the testing process.

Thanks Posted by Peskoe Ben(unfuddle username: bpeskoe)

[Battler45]

Ben, Vihar,

I have some questions regarding outstanding issues (geocodes and open houses details).

1. Geocodes. It looks like old updater doesn't get latitude and longtitude from MIBOR response. Isn't? I found in meta data some fields which probably can be used for this - DrivingLatitude and DrivingLongitude but I'm not sure. Anyway in my tests all MIBOR records don't return values for these fields. Also I've added functionality of geocode processing (similar to SoCal functionality) and it works.

2. Open-house updating. Can it be that MIBOR's user doesn't have some rights to get open house data? I've tried to get such data and I received error "User does not have access to Resource named OpenHouse".

Thanks Posted by Molchanov George(unfuddle username: george)

[Battler45]

Ben, Vihar,

I've completed latest minor changes (Driving Longitude / Driving Latitude, ListTypeDescrip).

Also I've added IndianapolisListingsPurger class to delete records but there is again known issue - User does not have access to Resource named DeletedListing.... Resource name is correct (according meta data), but it doesn't work. And we have not other ways to delete records as we getting only active - we can not get records with other statuses.

What we will do?

Thanks Posted by Molchanov George(unfuddle username: george)

[Battler45]

Ben,

Forgot to add one point - I haven't been able to verify the recent changes on the dev. version of weirproperties.com. For some reason, it redirects me to the live site. I commented out the corresponding Rewrite rules in the .htaccess file but it didn't have any effect. Pls let me know if I'm missing something.

Thanks. Posted by Shah Vihar(unfuddle username: vshah)

[Battler45]

Vihar,

On the RealEstat3000.com site, the home page carousel does reference /res/includes/listings-carousel-2.asp, but within that file is contained a reference to /property-search/widgets/carousel.asp and /property-search/widgets/carousel-close.asp. So, as long as the carousel on RealEstate3000.com home page works, I think we should be fine.

And I am fairly certain that carousel-listings.asp is not used. I had added that file originally, but at a later point realized that the display portion of each carousel is site specific, and so that is when I created the carousel.asp file, but I must have forgotten to remove the carousel-listings.asp file.

Thanks. Posted by Peskoe Ben(unfuddle username: bpeskoe)

[Battler45]

George,

Yes, it appears I made a mistake. I thought that we need to pull both Active and Pending listings for the MIBOR feed.

We only need to pull the Active listings.

Can you please update the MIBOR updater accordingly, and then run an update that will remove all Pending listings currently in the database?

Thanks,

Ben Posted by Peskoe Ben(unfuddle username: bpeskoe)

[Battler45]

George,

Re. small application (comment-#71) - I realize that there would thousands of records in the MLSPhotos table for a region so it won't be good idea to select all records in one-go. My thought is that it would better to process photos in batches of x number of records. I'm doing something similar in generating email-alerts. If you can take a look at the EmailAlertService.cs file, the SendAlertsInBatch method retrieves records in batch of 100 records. We can do something similar in moving photos to new folder structure. I think it would be good to process photos in batch of 1000 records.

In addition, it would be good to retrieve only unprocessed records. That is, when a photo is moved to new structure, the corresponding MLSPhotos record's PhotoServer field would be set to 1. So in retrieving batch of MLSPhotos records, we could add a filter to retrieve photos that haven't yet been processed (i.e. having PhotoServer = 0). This will allow us to deal with a situation where the application may have to be restarted / rerun.

Pls let me know if this sounds fine.

Thanks. Posted by Shah Vihar(unfuddle username: vshah)

[Battler45]

George,

To further clarify, the existing updater for mibor code will probably be confusing to you. It references the sys IDs of each database field, rather than the long field names, which is what you will be using when you query the RETS server using libRETS (I believe). You can use the T5 Field Info RETS Updated spreadsheet attached to the first comment in this thread to help you see which sys IDs match to which long field names.

In bringing any new data feed in line with the new updater system, the most difficult part will always be to determine how the fields from the host RETS database should be mapped into the fields of our own database.

In certain cases, such as when dealing with City, State, and Zip Code, these are straightforward.

However, in dealing with other types of listing data, such as interior and exterior features, lot features, etc., this can be more complicated. So, in working this out, referring to the existing SoCal MLS updater mappings (region 10) may also be quite helpful for you, because that region will (I believe) contain many of the same field names and will share many similar mappings.

Please let me know if you have any questions.

Thanks. Posted by Peskoe Ben(unfuddle username: bpeskoe)

[Battler45]

George,

The HomeToIndy.com client phoned today to report that her board has released a new class of data -- data for "Past Solds", or listings that have sold and are no longer on the market.

Can you pull the metadata for the MIBOR / Indy feed and see if there are new resources in the feed which would allow us to acccess that data?

Thanks. Posted by Peskoe Ben(unfuddle username: bpeskoe)

[Battler45]

George,

Ben would be able to provide more details about exact mappings for the above mentioned fields.

Ben, could you pls check the mapping file attached in comment-17 and let us know the mappings for the remaining Features fields?

Thanks. Posted by Shah Vihar(unfuddle username: vshah)

[Battler45]

Vihar,

Yes, that does sound like a good plan, please proceed.

Thanks. Posted by Peskoe Ben(unfuddle username: bpeskoe)

[Battler45]

George,

I've modified the MIBOR updater to ignore listings with "Public Internet" set to "No". I've pushed those changes and planning to run a full update of all Indy listings in the next 15 minutes. If you happen to check this before that can you pls confirm the changes?

Ben - removing the existing listings with the "Public Internet" set to "No" would be a bit tricky in that those listings are already added to the MLS table, and there is no other way to remove them other than removing those MLS records through the backend. I was thinking that it would be fine to execute the spRemoveListingByMLSNumber stored-procedure manually for all such listings. That is, after the full update is completed today, the DateLastUpdate date of all active listings having "Public Internet" set "Yes" would have changed. The DateLastUpdate field of the other listings, most likely having "Public Internet" set "No", would remain unchanged and so it would be fine to remove such listings by executing the spRemoveListingByMLSNumber stored-procedure manually. Pls let me know if you see any issue with this.

Thanks.

Posted by Shah Vihar(unfuddle username: vshah)

[Battler45]

Vihar, What do you mean by MIBOR meta-data? I'm using T5_field_info_RETS__Updated01-17-09.xls document attached in first post. Or you want to see how I perform this mapping (so I can give my code of "Parse.." function). Thanks Posted by Molchanov George(unfuddle username: george)

[Battler45]

George,

Forgot to add - you can check the database schema by remoting to the following machine:

IP: 75.103.119.195

User Name: vshah

Password: ViharShah$123

Pls note that this is development DB, and, has almost identical schema to the live DB. There are total five DBs on this server: sierrainteractive-1 to 4, and sierrainteractive-updater. Pls note that currently sierrainteractive-3 is being used by the live sites so pls make sure that you don't change anything in that database. Ideally, you would only be needing to refer to the sierrainteractive-4 and sierrainteractive-updater DBs.

Pls let me know in case of any issues.

Thanks. Posted by Shah Vihar(unfuddle username: vshah)

[Battler45]

George,

I've merged changes to the master branch and have pushed the same. Pls note that I had created a persistence method, GetAllSourceUniqueIds for SoCal updater. A similar method, GetAllSourceUniqueID was being used in the MIBOR / GLAR updaters. I've removed the GetAllSourceUniqueID method and have changed references appropriately.

Thanks. Posted by Shah Vihar(unfuddle username: vshah)

[Battler45]

Ben,

RE: #66: I've restarted the DEDN173 server but I'm now unable to remote into it. The server appears to have restarted correctly as I could access the CAA website. Could you pls look into the same?

As for the size of SoCal photos, it is definitely lot more when compared to other regions. I verified that all these photos are associated with active listings. Moreover, the ratio of photos added to photos removed is more than 2 majority of the times so this size in all likelihood will increase.

Thanks.

Posted by Shah Vihar(unfuddle username: vshah)

[Battler45]

Vihar & George,

I just realized that we will, of course, need to re-load all photos because the photos will all be located on a different server, and the folder structure used by the new updater is different than that of the current updater.

Vihar, the problem with this is that we can't just delete the existing region 25 listings from the database, because all of our lead records reference the IDs of these listings in the database. Really, all we need to do is update the MLSPhotos records for each listing -- everything else can stay the same, right?

Do you have any thoughts on what might be the best way to make that transition?

Thanks Posted by Peskoe Ben(unfuddle username: bpeskoe)

[Battler45]

Ben,

I've checked MIBOR metadata and I don't see there "Past Solds" class. We have access to the next classes:

Residential Lease (not imported)

ResidentialProperty (imported as Residential)

Commercial Retail (not imported)

LotsAndLand (imported as LotsAndLand)

Commercial/Industrial (not imported)

Commercial Office (not imported)

Commercial Multi-Family (not imported)

Farms & Agricultural Land (not imported)

MultiFamily (imported as MultiFamily)

Commercial Land (not imported)

CommonInterest (not imported)

Thanks Posted by Molchanov George(unfuddle username: george)

[Battler45]

Vihang,

I've already done this -- updated the dev sites on DEDN173 to pull content from the live db rather than the development db (I need to work on a new site today).

Thanks Posted by Peskoe Ben(unfuddle username: bpeskoe)

[Battler45]

Ben,

Thanks. The changes to .htaccess have now taken effect. Pls note that I've reset the passwords for the 'n3wt0n' and 's13rra' for consistency.

Pls let me know your inputs on comment #99 when you have chance.

Thanks. Posted by Shah Vihar(unfuddle username: vshah)

[Battler45]

George,

I'll merge those changes to the master branch later. There may be some conflict in merging those changes so I'll work on it after finishing my current task. I believe your current work shouldn't be affected by the merge. Pls continue to work / push changes to the master branch.

Thanks. Posted by Shah Vihar(unfuddle username: vshah)

[Battler45]

George,

In implementing new updater system for region-25, you might want to refer to the existing updater system implementation (especially for field mappings). I've attached a zip file containing the source-code for the same. It contains the following folders:

mibor - contains code specific to updater-system for region 25. The "mls_autoupdates.asp" file contains the entry-point; that is, the programming in this file will initiate the updates for region-25. It will call appropriate functions like UpdateListings, RemoveOldListings, UpdatePhotos etc... for performing various update steps. Some of these functions could be defined in the mls_functions.asp file present under this folder. And, rest of the functions would be defined in the common code.

res/includes - contains code common to all updater-systems. The mls_functions.asp file for example defines InsertUpdateMLSListing function, which is called to add/update listing records.

---

I'm not sure if you had a chance to work with libRETS library. If you haven't, I would recommend first creating a sample project that uses the libRETS library. Pls use the above RETS credentials for connecting to the MIBOR RETS server. Ideally, you should first download the metadata from this server. That would help you in mapping listing fields.

Pls note that the MIBOR region data is stored in the sierrainteractive-1 database. You can copy the latest back up of the database (development) from the following FTP location:

Host: 75.103.116.57 User-name: george_dev_db Password: GeorgeM$123

Since the development DB is has restricted access, pls restore the above back-up on your local DB and configure the updater code to work with your local DB.

Pls let me know if you have any queries.

Thanks.

Posted by Shah Vihar(unfuddle username: vshah)

[Battler45]

Ben,

Re. #68 - old and new updaters call the same stored-procedure, spUpdateMLSPhoto for updating a photo-record. So perhaps that sp can be updated to set changed-date in the DateAdded field. Pls let me know if this sound fine. I'll update the sp.

Thanks. Posted by Shah Vihar(unfuddle username: vshah)

[Battler45]

George,

Sorry for the delay in reply. I agree that Standard name is missing in several fields. I guess we could make use of other descriptive fields like short-name / description to get info. about fields. I've downloaded RETS metadata, which includes such description values. Pls find attached the file listing the entire MIBOR RETS metadata. Pls note that we would mainly be interested in "Property" resources (line# 4092). Each "Class Name" under the "Property" resource would represent type of listing e.g. Residential Lease, Residential/Condo, Commercial Retail, Commercial Land and so on...

The pattern for class-name is as follows: Class name: class-description (class-system-name) [class-standard-name]. For example, Class name: Residential Lease(1) [], Class name: Vacant Lots/Land(4) [LotsAndLand]

The pattern for table-name is as follows: Table name: table-short-name (table-system-name) [table-standard-name]. For example, Table name: Base Desc(21) [], Tot FB(24) [BathsFull].

You can use this information for mapping sierrainteractive db fields to corresponding RETS fields. In addition, you can use the system-name / standard-name for querying RETS data-feed.

As for finding recently updated records, pls use the ChgDt (151) (line# 4690) value to query RETS server.

Pls let me or Ben know if you have additional queries in mapping other fields (especially composite fields)

Thanks.

Posted by Shah Vihar(unfuddle username: vshah)

[Battler45]

RETS meta-data Posted by Molchanov George(unfuddle username: george)

[Battler45]

Vihar,

I think it is fine idea to host images on 3rd-party service.

Also I want to notice that MIBOR updater is not completed as I didn't test open-houses updating.

Thanks Posted by Molchanov George(unfuddle username: george)

[Battler45]

Ben,

Assuming we will be continuing with the current setup of hosting images on the Crystaltech cloud server, there are a couple of approaches for moving MIBOR images:

1 -- to re-download all photos through the new updater system. This will require minimum change in that we would need to update the photo-url generation to take into account active photo-server value (as described in comment #52). This will however be fairly time-consuming process as all photos would have to be re-downloaded from the RETS server and re-processed.

2 -- to write a small application, which will copy processed images from the old photo-server and update the database records with new path value. This should be faster compared to the 1st approach in that we'll be using FTP interface for downloading photos from another Crystaltech machine.

Alternatively, all images can be copied manually from the old server to the Crystaltech cloud server, and, then the new application should be run to update db records. This processing would be fairly quick in that the processing would be done locally. However, the time to FTP all images to new server might be high assuming large number of images would need to be transferred.

In either case, the new application will be useful in moving photos of other regions as and when we convert them to new updater system.

My thought is that regardless of the approach followed, we should add a new field, PhotoServer (comment #52), and update the photo-url generation logic. This will ensure that photos are continued to be served correctly until all photos have been moved to the Crystaltech cloud server.

Pls let me know your thoughts / suggestions.

Thanks.

Posted by Shah Vihar(unfuddle username: vshah)

[Battler45]

George,

I've made changes to the image-processing code to generate large images in 640x640 dimensions. I've pushed this change to the master branch. Can you pls fetch the latest code and update your local code?

This just for your info - I've created a new branch, production in the updater git repository. This branch will contain code that is currently used for production updater service. That is, it won't contain any development specific code. I'll maintain this branch to make sure that development specific changes are not pushed to the production service. Pls continue pushing your changes to the master branch (which would contain all development specific changes like you are making for implementing MIBOR updater).

As for photo-download/process testing - I would suggest you to first test it on your local machine (if you haven't already). For this, pls set the LastUpdateTimeStamp for photo-step (16) to current date. This will make sure that only most recent photos are downloaded/processed. Once you've verified the changes, pls test the MIBOR updater on the development server. If everything is working as expected, we can then download all records / photos (by setting the LastUpdateTimeStamp value to null for all steps). Pls let me know your thoughts.

Thanks.

Posted by Shah Vihar(unfuddle username: vshah)

[Battler45]

George,

I tested the MIBOR updater on local machine. Overall, it appears to be working fine. There are a few points that might require changes:

1 -- Looks like the DrivingLongitude & DrivingLatitude values are not available for majority of the listings so we would have to geo-code all listings through the Geocoder step. In that case, I think, the code, which processes latitude / longitude values (last few lines in the IndianapolisListingsImporterForProperty.cs file) can be removed.

2 -- The ListType and ListTypeDescrip values are not correctly set. The ListType value should reflect the type of listing (i.e. 1 for residential, 4 for land/lot, 5 for multi-family and so on...). The ListTypeDescrip value should reflect sub-type (i.e. condominium, residential, single lot and so on...)

Ben - I checked the meta-data for the MIBOR region but couldn't find any suitable field, which could be holding sub-type description. I've attached a zip file containing meta-data / actual data for your reference. Pls let me know the field, which can be used for populating ListTypeDescrip value.

3 -- Per the current implementation, importer downloads all listings regardless of their status. Ideally, it should only process listings having status Active / Pending. There is a field, Status (134) which can be used for retrieving Active / Pending listings only.

Ben - Strangely though when I queried the RETS server to get listings filtered by change-date, it returned active / pending listings only. Perhaps listings with other status might not have been changed. Per the meta-data, a listing can have one of the following statuses: Active, Expired, Leased, Pending, Released, Sold or Withdrawn. Do you think it would be fine to ignore the listing status?

4 -- In the live database, each section in the Featured* field is enclosed within

. For example,

FeaturesBasement -

Basement: [Actual Values]

FeaturesConstruction -

Construction Stage: [Actual Values]

Construction Type: [Actual Values]

Construction Status: [Actual Values]

Pls change the MIBOR updater to set Features* values accordingly.

5 -- Per the current implementation of photo-downloader, if a listing already has one or more photos in the database then all those photos are removed and re-downloaded. I think this might increase the time to download / process all photos. I guess it should be alright to remove all photos only if the number of photos on the RETS server is less compared to the number of photos in the database. Pls let me know if you see any issue with this.

6 -- The only missing step (other than the OpenHouse step) is removal / purging of old listings. As and when listings are sold / withdrawn, they are made unavailable from the REST server (either physically or by changing their status). We need to take this into account in synchronizing RETS data to our database. So an additional step will need to be implemented (in line with the SoCal listing purger step (18)), which will process such listings and remove those from our database. Can you pls review the SoCalListingListingPurger class and implement similar step for MIBOR region?

Pls let me know if you have any queries.

Thanks.

Posted by Shah Vihar(unfuddle username: vshah)

[Battler45]

George,

Per Ben's feedback,

1 -- the current format of Features* value is fine. Pls ignore my comment.

2 -- the approach to remove all photos and re-download them sounds fine for MIBOR region. In SoCal RETS server, we do have access to Media resource, and, therefore, we could make use of the modified photos' details. Unfortunately, we don't have access to Media resource in MIBOR RETS server so pls continue you with your approach of deleting and re-downloading all imagescomment).

3 -- since we have access to only Active / Pending listings no need to make any changes for the same. The MIBOR importer is already populating the Status field (MLS table) with appropriate value from the RETS field, 134 (Status) so no further change needs to be made in this regard.

Pls make changes for the other points discussed above. Pls let me know if you have any queries.

Also, in reviewing the photo-processing code I realized that it is the same for both regions. Therefore, I've renamed one of the PhotoProcessor classes to DefaultPhotoProcessor and removed the other. As with DefaultGeocoding, we can continue using this class for other regions unless there is any change in the approach.

Thanks.

Posted by Shah Vihar(unfuddle username: vshah)

[Battler45]

Vihar,

I thought scripts under Updater\Sierra.Updater.Database folder are intended only for sierrainteractive-updater database. Isn't? And my script has to be run on sierrainteractive-n databases.

Also I attach script here for your testing while it is not committed.

Thanks Posted by Molchanov George(unfuddle username: george)

[Battler45]

Vihar,

Sounds great, thanks.
Posted by Peskoe Ben(unfuddle username: bpeskoe)

[Battler45]

Vihar & George,

Can you update me on the status of this? Has the updater completed and have the indicated listings been removed from the site? I do see that the original listing MLS # 21163069 is no longer available, but wanted to confirm before responding to the MIBOR contact.

He also just emailed to request that we verify we are not displaying the address for listings where the "Address yes/no" field is set to "No" -- I believe we are changing the address on these listings to "000 Confidential" but can you double-check that for me as well?

Thanks. Posted by Peskoe Ben(unfuddle username: bpeskoe)

[Battler45]

George,

I've logged off from the server. Can you pls try after 5 mins and let me know if you can log into the server?

As for the changes to be applied to your local copy - you'll need to merge the remote changes into your local branch. This can be done by executing the following command git merge origin master. I'm not sure if origin is the name of your remote branch. You can check the same by executing git branch -r command; this will list all remote branch maintained by the git repository.

Alternatively, you can run git pull origin master. This will first fetch remote changes and immediately merge them in your local repository. So in a way it does the job of git fetch and git merge.

Pls let me know if you have any queries.

Thanks Posted by Shah Vihar(unfuddle username: vshah)

[Battler45]

Ben,

Yes, I made changes within the Inetpub-LIVE folder. To be specific, I wanted to check the old list / detail pages for photos. In doing that, I made changes so that photo urls were invalid. I then checked those pages expecting them to not show photos but photos were displayed correctly. Perhaps it may have something to do with caching; strangely though photos were not rendered in hometoindy list page. I'll spend more time on this later today.

Thanks Posted by Shah Vihar(unfuddle username: vshah)

[Battler45]

George & Vihar,

RE: #69, I've contacted MIBOR representatives to see if they can give us access to the requested resources. I will let you know.

Thanks. Posted by Peskoe Ben(unfuddle username: bpeskoe)

[Battler45]

Vihar,

RE: #63 - Your alternate approach does sound good and workable. I guess that appending a querystring parameter to the photo filename does not impact the ability of the browser to render the images? (I've tested, and it does not seem to be a problem.) In that case, this does seem like the best and easiest solution.

I have not yet checked when and how the DateAdded field in the MLSPhotos table is updated, but I will do so tomorrow, and will make sure that the old updater is set to update this field appropriately. Can you check the new updater to make sure it is also updating this value appropriately?

Thanks Posted by Peskoe Ben(unfuddle username: bpeskoe)