search

Battler45 / SierraMigrationToGitHub

Migration from unfuddle to GitHub
0 stars 0 forks source link

Convert Region 25 (MIBOR / Indy) to new updater #216

Open Battler45 opened 4 years ago

Battler45 commented 4 years ago

George,

The first project on which you'll be working will be to convert an existing MLS data feed to the new updater system.

Vihar will be able to provide you with the technical details for installing GitHub and downloading a local copy of the current code base.

Vihar, will you also please provide George with whatever server and / or database access he needs for this initial project? Please share with him any necessary logins and create any new users as necessary. Also, I think that converting region 25 will be a good place to start because it shares the same MLS data provider (Marketlinx) as the one you have already converted (SoCal MLS). You can find my existing code in:

C:\Inetpub\mls4.sierrainteractivemls.com\admin\mibor

on DEDQ61

The login credentials to the RETS server are:

  • UserID: 31041rets

  • Password: BENJ1001

  • UserAgent: sierrainteract

  • Password: h1ghplateau

RETS URL: http://inr.rets.interealty.com/Login.asmx/Login

As you'll see when you examine my existing code, I was using the sysid name for the database fields rather than the long names for the fields (which were being used for SoCal MLS). I did this because I had a lot of difficulty with the RETS Connector software that I was using to download the daily data files. To help translate between the two, I've attached a spreadsheet provided to me by MIBOR (which stands for Metropolitan Indianapolis Board of Realtors) that cross-references the long field names with their numeric IDs.

Please let me know if you have any questions.

Thanks.

Posted by Peskoe Ben(unfuddle username: bpeskoe)

Battler45 commented 4 years ago

George,

Forgot to add - you can check the database schema by remoting to the following machine:

IP: 75.103.119.195

User Name: vshah

Password: ViharShah$123

Pls note that this is development DB, and, has almost identical schema to the live DB. There are total five DBs on this server: sierrainteractive-1 to 4, and sierrainteractive-updater. Pls note that currently sierrainteractive-3 is being used by the live sites so pls make sure that you don't change anything in that database. Ideally, you would only be needing to refer to the sierrainteractive-4 and sierrainteractive-updater DBs.

Pls let me know in case of any issues.

Thanks. Posted by Shah Vihar(unfuddle username: vshah)

Battler45 commented 4 years ago

George,

I've set you up so that you should be receiving updates on this task / thread -- please confirm. This thread contains the initial details you will need to get started on your first task, and Vihar can provide you with the background, technical details, etc. that you will need to get started.

Please let me know if you have any questions.

Thanks. Posted by Peskoe Ben(unfuddle username: bpeskoe)

Battler45 commented 4 years ago

Vihar,

RE: #99

Sorry for missing this comment yesterday:

Thanks. Posted by Peskoe Ben(unfuddle username: bpeskoe)

Battler45 commented 4 years ago

Ben, It is very helpful! Thank you!

Posted by Molchanov George(unfuddle username: george)

Battler45 commented 4 years ago

George,

Sorry for the delay getting back to you on this -- I am working on it now and will have it for you by the end of the evening (my time). In the meantime, if you have more time to work on the project, perhaps you could begin looking into the photo downloads for this region? That (hopefully) should be very similar to the way it is currently being done for SoCal MLS / Region 10.

Thanks. Posted by Peskoe Ben(unfuddle username: bpeskoe)

Battler45 commented 4 years ago

George,

My apologies for the delay in responding you. I've verified the changes for ListType and SourceUniqueID; they are working fine.

Can you pls let me know if you have begun working on the photo-migration application?

Thanks. Posted by Shah Vihar(unfuddle username: vshah)

Battler45 commented 4 years ago

George,

I've logged off from the server. Can you pls try after 5 mins and let me know if you can log into the server?

As for the changes to be applied to your local copy - you'll need to merge the remote changes into your local branch. This can be done by executing the following command git merge origin master. I'm not sure if origin is the name of your remote branch. You can check the same by executing git branch -r command; this will list all remote branch maintained by the git repository.

Alternatively, you can run git pull origin master. This will first fetch remote changes and immediately merge them in your local repository. So in a way it does the job of git fetch and git merge.

Pls let me know if you have any queries.

Thanks Posted by Shah Vihar(unfuddle username: vshah)

Battler45 commented 4 years ago

Ben,

Was about to update you on this!! There were some issues in re-running full updates of MIBOR records. It failed thrice with the "Error occurred in logging into inr.rets.interrealty.com server" message. I'm not sure if this had anything to do with the full updates. Moreover, doing a full-update would have consumed a lot of time (approx. 8-10 hours) therefore I aborted the full update and ended up choosing a different approach.

I've modified the Indianapolis listing removal functionality to remove all listings that don't have "Public Internet" set to "Yes". This will not require manual removal of listings. In addition, it will also remove photo files from the server. The modified removal-functionality appears to have removed all listings having "Public Internet" set to "No". Pls check the same and let me know in case of any issues.

As for the point about not displaying address - I verified that we've programming in place to change the address to "000 Confidential Ave." for listings where the "Address yes/no" field is set to "No". There are total 137 listings in the database having this address.

Thanks. Posted by Shah Vihar(unfuddle username: vshah)

Battler45 commented 4 years ago

George,

In implementing new updater system for region-25, you might want to refer to the existing updater system implementation (especially for field mappings). I've attached a zip file containing the source-code for the same. It contains the following folders:

mibor - contains code specific to updater-system for region 25. The "mls_autoupdates.asp" file contains the entry-point; that is, the programming in this file will initiate the updates for region-25. It will call appropriate functions like UpdateListings, RemoveOldListings, UpdatePhotos etc... for performing various update steps. Some of these functions could be defined in the mls_functions.asp file present under this folder. And, rest of the functions would be defined in the common code.

res/includes - contains code common to all updater-systems. The mls_functions.asp file for example defines InsertUpdateMLSListing function, which is called to add/update listing records.

I'm not sure if you had a chance to work with libRETS library. If you haven't, I would recommend first creating a sample project that uses the libRETS library. Pls use the above RETS credentials for connecting to the MIBOR RETS server. Ideally, you should first download the metadata from this server. That would help you in mapping listing fields.

Pls note that the MIBOR region data is stored in the sierrainteractive-1 database. You can copy the latest back up of the database (development) from the following FTP location:

Host: 75.103.116.57 User-name: george_dev_db Password: GeorgeM$123

Since the development DB is has restricted access, pls restore the above back-up on your local DB and configure the updater code to work with your local DB.

Pls let me know if you have any queries.

Thanks.

Posted by Shah Vihar(unfuddle username: vshah)

Battler45 commented 4 years ago

Vihar,

I've fixed issues with ListType and SourceUniqueID.

Thanks Posted by Molchanov George(unfuddle username: george)

Battler45 commented 4 years ago

George & Vihar,

I do think it might be worthwhile to develop a small application which would compare data rows between dev and live DBs for the same listing to make sure that the data is the same -- mainly because we do have a number of additional regions to convert from the old updater to the new updater, and so this application would be used for all of those regions and would therefore save us quite a bit of time in testing.

Unless, Vihar, you have already written some scripts that will do this type of comparison or have something else in mind?

Thanks. Posted by Peskoe Ben(unfuddle username: bpeskoe)

Battler45 commented 4 years ago

Status of task: I'm able to get response from RETS server and going to parse it.

Ben, I have question about query for request: which date should be used in query to select only outstanding records? For example table "RES" has several date fields - "List Date", "Entry Date" and "Change Date". Do you know which is correct? Posted by Molchanov George(unfuddle username: george)

Battler45 commented 4 years ago

George,

I've modified the MIBOR updater to ignore listings with "Public Internet" set to "No". I've pushed those changes and planning to run a full update of all Indy listings in the next 15 minutes. If you happen to check this before that can you pls confirm the changes?

Ben - removing the existing listings with the "Public Internet" set to "No" would be a bit tricky in that those listings are already added to the MLS table, and there is no other way to remove them other than removing those MLS records through the backend. I was thinking that it would be fine to execute the spRemoveListingByMLSNumber stored-procedure manually for all such listings. That is, after the full update is completed today, the DateLastUpdate date of all active listings having "Public Internet" set "Yes" would have changed. The DateLastUpdate field of the other listings, most likely having "Public Internet" set "No", would remain unchanged and so it would be fine to remove such listings by executing the spRemoveListingByMLSNumber stored-procedure manually. Pls let me know if you see any issue with this.

Thanks.

Posted by Shah Vihar(unfuddle username: vshah)

Battler45 commented 4 years ago

Ben,

Thanks. The changes to .htaccess have now taken effect. Pls note that I've reset the passwords for the 'n3wt0n' and 's13rra' for consistency.

Pls let me know your inputs on comment #99 when you have chance.

Thanks. Posted by Shah Vihar(unfuddle username: vshah)

Battler45 commented 4 years ago

Ben,

I've added a couple of new functions at the bottom of Inetpub_LIVE\Sites\MLS_Tools_Integrated\res\includes\functions.asp. These functions are used for forming photo-urls based on the PhotoServer value. In order to test these functions I had switched the condition to generate photo-urls based on the new format.

That said, I realize that I had overlooked the fact that several regions don't have PhotoPath value, and, therefore, photo-url format would be the same regardless of the PhotoServer value. This would have been the case for schulerbauer as photos for regions 3 / 4 don't have PhotoPath values. I think we are fine in this case. My apologies for the oversight.

Thanks. Posted by Shah Vihar(unfuddle username: vshah)

Battler45 commented 4 years ago

Ben, Vihar,

I completed test of MIBOR updater on development server. Update process was executed without errors. I don't understand how to test results using of comparison of live (www.hometoindy.com) and development (hometoindy.sierrainteractivedev.com) sites. I see that some record is added into database - I can find this record by its MLS number in database sierrainteractive-1. But when I'm trying to find this records on sites (with using of search by MLS number) I could not get any results. Does development site use database sierrainteractive-1 on sql4.sierrainteractivemls.com indeed? Anyway I'm not sure that visual checking will give an opportunity to find out all issues (if these exist, I hope new updater is working without issues). Perhaps it will be more useful to make a small application intended to compare databases?

Thanks Posted by Molchanov George(unfuddle username: george)

Battler45 commented 4 years ago

Ben,

I've made changes to the production and dev DBs to add a new column, PhotoServer (bit) to the MLSPhotos table. When 0, it will indicate photos hosted on the old server, and 1 will indicate photos hosted on the cloud server. Currently its value is set to 1 for all records in the sierrainteractive-4 DB and 0 for all other records.

As mentioned in the above comment, the photo-migration app is working fine so we can now begin working on transitioning the MIBOR updater and its photos to the cloud server. I'll test the MIBOR updater on development server tomorrow; will update you on the same afterwards.

Also, we can now update the programming to build photo-urls to use the PhotoServer value. I think it would be a good idea to create a common function, which can then be called from all pages for displaying photos. My understanding is that the following pages will need to be updated:

Pls let me know your thoughts. I'll also update the new CAA accordingly.

Thanks.

Posted by Shah Vihar(unfuddle username: vshah)

Battler45 commented 4 years ago

Ben,

RE: #66: I've restarted the DEDN173 server but I'm now unable to remote into it. The server appears to have restarted correctly as I could access the CAA website. Could you pls look into the same?

As for the size of SoCal photos, it is definitely lot more when compared to other regions. I verified that all these photos are associated with active listings. Moreover, the ratio of photos added to photos removed is more than 2 majority of the times so this size in all likelihood will increase.

Thanks.

Posted by Shah Vihar(unfuddle username: vshah)

Battler45 commented 4 years ago

Ben,

I've updated dev. version of search-tools to use PhotoServer value in forming photo-url. I've also updated & verified a couple pages of live sites on DEDN173 for the same. I'll complete the remaining changes tomorrow. Several widget pages (\MLS_Tools_Integrated\widgets) have been updated but I'm not sure which pages embed those widgets. Could you pls let me know how I can check the following widgets:

Moreover, several xml feeds have also been updated. Is there any specific way to check the following feeds?

Pls let me know your inputs.

Thanks. Posted by Shah Vihar(unfuddle username: vshah)

Battler45 commented 4 years ago

Vihar,

Populating SourceUniqueID into current listings will not be a problem. I will re-run the MIBOR update for today on the old updater and modify it to include this field for all listings. I will let you and George know as soon as that is done.

Thanks. Posted by Peskoe Ben(unfuddle username: bpeskoe)

Battler45 commented 4 years ago

George,

Sorry for not being specific. I actually meant the files present under the Updater\Sierra.Updater.Database folder. I believe you've already made changes to the "scheama changes.sql" file to associate steps for MIBOR region. Pls add the new field statement to that file.

Thanks. Posted by Shah Vihar(unfuddle username: vshah)

Battler45 commented 4 years ago

George,

I tested the MIBOR updater on local machine. Overall, it appears to be working fine. There are a few points that might require changes:

1 -- Looks like the DrivingLongitude & DrivingLatitude values are not available for majority of the listings so we would have to geo-code all listings through the Geocoder step. In that case, I think, the code, which processes latitude / longitude values (last few lines in the IndianapolisListingsImporterForProperty.cs file) can be removed.

2 -- The ListType and ListTypeDescrip values are not correctly set. The ListType value should reflect the type of listing (i.e. 1 for residential, 4 for land/lot, 5 for multi-family and so on...). The ListTypeDescrip value should reflect sub-type (i.e. condominium, residential, single lot and so on...)

Ben - I checked the meta-data for the MIBOR region but couldn't find any suitable field, which could be holding sub-type description. I've attached a zip file containing meta-data / actual data for your reference. Pls let me know the field, which can be used for populating ListTypeDescrip value.

3 -- Per the current implementation, importer downloads all listings regardless of their status. Ideally, it should only process listings having status Active / Pending. There is a field, Status (134) which can be used for retrieving Active / Pending listings only.

Ben - Strangely though when I queried the RETS server to get listings filtered by change-date, it returned active / pending listings only. Perhaps listings with other status might not have been changed. Per the meta-data, a listing can have one of the following statuses: Active, Expired, Leased, Pending, Released, Sold or Withdrawn. Do you think it would be fine to ignore the listing status?

4 -- In the live database, each section in the Featured* field is enclosed within

. For example,

FeaturesBasement -

Basement: [Actual Values]

FeaturesConstruction -

Construction Stage: [Actual Values]
Construction Type: [Actual Values]
Construction Status: [Actual Values]

Pls change the MIBOR updater to set Features* values accordingly.

5 -- Per the current implementation of photo-downloader, if a listing already has one or more photos in the database then all those photos are removed and re-downloaded. I think this might increase the time to download / process all photos. I guess it should be alright to remove all photos only if the number of photos on the RETS server is less compared to the number of photos in the database. Pls let me know if you see any issue with this.

6 -- The only missing step (other than the OpenHouse step) is removal / purging of old listings. As and when listings are sold / withdrawn, they are made unavailable from the REST server (either physically or by changing their status). We need to take this into account in synchronizing RETS data to our database. So an additional step will need to be implemented (in line with the SoCal listing purger step (18)), which will process such listings and remove those from our database. Can you pls review the SoCalListingListingPurger class and implement similar step for MIBOR region?

Pls let me know if you have any queries.

Thanks.

Posted by Shah Vihar(unfuddle username: vshah)

Battler45 commented 4 years ago

Hi Vihar, Could you please explain more detailed how I can test the application? As I understand the specified RETS servers are live not "sandbox"? Is it ok to connect to servers for testing purposes? Thanks Posted by Molchanov George(unfuddle username: george)

Battler45 commented 4 years ago

Ben,

Re. #68 - old and new updaters call the same stored-procedure, spUpdateMLSPhoto for updating a photo-record. So perhaps that sp can be updated to set changed-date in the DateAdded field. Pls let me know if this sound fine. I'll update the sp.

Thanks. Posted by Shah Vihar(unfuddle username: vshah)

Battler45 commented 4 years ago

Vihar,

I implemented the new approach to delete listings of MIBOR region. It works pretty fast (about 28 thousand records from MIBOR RETS server). All is committed excepting sql script with stored procedure for sierrainteractive-n databases (it is attached).

Thanks Posted by Molchanov George(unfuddle username: george)

Battler45 commented 4 years ago

George,

I tested the latest changes. The ListTypeDescrip value is correctly populated now but the ListType value doesn't reflect the type of listing. It should be 1 for residential, 4 for land / lots and 5 for multi-family. Currently, it set to 1 for all types of listings. Pls make necessary change.

Also, I think we may need to change the way SourceUniqueID field's value is set. Currently the ListingKey (150) value is assigned to this field. I think this value may not be unique in the MIBOR RETS server. Can you pls look into assigning "sysid" value to the SoruceUniqueID field? The SourceUniqueID field should store the unique value, which can be used to identify record in the MIBOR database (kind of primary-key). This value is used mainly in checking physically removed listings (In the PurgeListing step (18)).

Thanks. Posted by Shah Vihar(unfuddle username: vshah)

Battler45 commented 4 years ago

Vihar,

I'm working now on the photo-migration tool. I hope the first revision will be ready today.

Thanks Posted by Molchanov George(unfuddle username: george)

Battler45 commented 4 years ago

Ben, Vihar,

I tested photos downloading ang calculation of geocodes on development server. It works.

Thanks Posted by Molchanov George(unfuddle username: george)

Battler45 commented 4 years ago

Ben,

Yes, I made changes within the Inetpub-LIVE folder. To be specific, I wanted to check the old list / detail pages for photos. In doing that, I made changes so that photo urls were invalid. I then checked those pages expecting them to not show photos but photos were displayed correctly. Perhaps it may have something to do with caching; strangely though photos were not rendered in hometoindy list page. I'll spend more time on this later today.

Thanks Posted by Shah Vihar(unfuddle username: vshah)

Battler45 commented 4 years ago

Vihar,

Would you mind taking a look at the data on the development server, including the photos and geocodes, and testing to make sure it matches the live region 25 data?

If so, then I suppose we don't need to re-download all photos and geocodes for all Indy listings -- could we just use the new updater system to pick up from where the old updater system leaves off -- perhaps we could schedule that for Thursday morning this week if everything looks good?

Thanks

Posted by Peskoe Ben(unfuddle username: bpeskoe)

Battler45 commented 4 years ago

Ben,

I've tested the MIBOR updater on the development server. It worked fine. I guess we can now transition the MIBOR updater to the new updater. However, we'll first need to migrate existing photos to the cloud server. Pls let me know your inputs on comment#93.

Thanks Posted by Shah Vihar(unfuddle username: vshah)

Battler45 commented 4 years ago

George & Vihar,

RE: #69, I've contacted MIBOR representatives to see if they can give us access to the requested resources. I will let you know.

Thanks. Posted by Peskoe Ben(unfuddle username: bpeskoe)

Battler45 commented 4 years ago

George,

Yes, the specified RETS servers are live. I think it should be ok to connect to those servers for testing purpose.

I understand that you want to test the new updater system - the SoCal updater system to be specific. To do so, you can run the WindowsServiceTest console application. The UpdateProcessor.cs file in this project contains the entry-point function, PerformUpdatesForRegion. This function will execute appropriate updater steps (depending on the steps supported for a region - as defined in the Updater_MLSRegionUpdateSteps table). If you want to test a specific step, pls comment out appropriate code in the PerformUpdatesForRegion function.

Also, you would need to change certain configuration parameters defined in the app.config file. Some of these parameters are:

Pls let me know if you need additional info.

Thanks.

Posted by Shah Vihar(unfuddle username: vshah)

Battler45 commented 4 years ago

George,

Just checking in to see how things are coming -- were my last comments helpful, and have you been able to make further progress?

Thanks. Posted by Peskoe Ben(unfuddle username: bpeskoe)

Battler45 commented 4 years ago

George,

As you have time can you pls work on building a small application for migrating photos from old-updater to new-updater? As discussed above, the old-updater server's performance is not always good and so when a region is converted to the new updater system, photos of that region should be moved to the Crystaltech cloud server.

As far as photo processing is concerned, the main difference between two updater systems is the way photos are organized. As you already know, each photo is resized in three different sizes i.e. thumbnail , regular and large. These photos are organized under the mlsthumbs, mlspics and mlslarge folders respectively. The actual organization however varies for some regions. Some of the recently added regions support organization in sub-folders. As of now the following three regions support organization by sub-folders - 10 (SoCal), 24 (Denver) and 25 (MIBOR); for rest of the regions photos are organized under single folder.

To give you an idea about the folder organization on the existing photo serve:

Root-Photo-Folder

    3
      mlsthumbs
      mlspics
      mlslarge

    ...

    25_1
      mlsthumbs
      mlspics
      mlslarge

     ...

    25_100
      mlsthumbs
      mlspics
      mlslarge
...

This structure has been tweaked for new updater system as follows

Root-Photo-Folder

  10
    mlsthumbs
        AA
        BB

        ...

    mlspics
        AA
        BB

        ...

    mlslarge
        AA
        BB

        ...

Going forward all regions will need to conform to the new photo-organization approach (as implemented in the new updater system). This will be done on region-by-region basis as and when regions are converted to the new updater system.

So the goal of the new application would be to organize photos per the new approach. It will have to deal with two types of existing organization

For start, you can implement the application to process MIBOR photos and move them per new organization. It can later be refined to support all regions.

As and when photos are moved to new organization, we would have to set certain flag in the corresponding MLSPhotos record so that the front-end site builds correct photo-url. As discussed in the comment-#52, a new field PhotoServer would need to be added to the MLSPhotos table. When a photo is organized per new approach, the corresponding PhotoServer value would be set to 1 so that the front-end site builds appropriate url.

My thought is that the application can be implemented to perform the following steps

1 -- to read photo records from the MLSPhotos table for MIBOR region (25).

2 -- it will then copy the corresponding photos (thumbnail, regular and large) from the old folder-structure and move to new structure. Pls use the logic of generating sub-folder name from the PhotoDownloader code (GetHashedDirectoryName function).

3 -- if successful, the corresponding PhotoServer flag should be set to 1.

Since the goal of the application is to support all regions, a configurable parameter should be added to indicate the region to process. A couple of additional parameters can be used for setting source / destination folders. Also, it would be fine to create a new console application (similar to the updater console application) under the updater solution so that it can utilize the existing framework / source-code where possible.

Pls let me know if you have any queries / suggestions.

Thanks.

Posted by Shah Vihar(unfuddle username: vshah)

Battler45 commented 4 years ago

George,

I see -- to give you a bit of background, there are a number of different data providers around the United States. These are companies that build and maintain the software for individual boards of realtors. The Socal MLS and the Indianapolis MLS (MIBOR) both use a company called MarketLinx.

This means that, although some of the fields and field names may vary between SoCal MLS and MIBOR MLS, most of them will be the same, and the way you query the database for MIBOR will be very similar to the way the database is queried for the existing SoCal MLS updater, and so you should feel free to look at SoCal MLS updater as a reference for MIBOR. (Other boards of realty use different data providers, and so this will not always be the case.)

That said, I am not sure of the answer to your question, but I believe Vihar should be able to answer this for you, since I am sure it is the same in MIBOR as it is in SoCal MLS. (My guess is that "Change Date" is the correct answer, but I am not certain.)

Ben Posted by Peskoe Ben(unfuddle username: bpeskoe)

Battler45 commented 4 years ago

Ben, Vihar,

I've completed latest minor changes (Driving Longitude / Driving Latitude, ListTypeDescrip).

Also I've added IndianapolisListingsPurger class to delete records but there is again known issue - User does not have access to Resource named DeletedListing.... Resource name is correct (according meta data), but it doesn't work. And we have not other ways to delete records as we getting only active - we can not get records with other statuses.

What we will do?

Thanks Posted by Molchanov George(unfuddle username: george)

Battler45 commented 4 years ago

George,

1 - Yes, it's fine to geocode listings using our own functionality -- that's what I am also doing in the current updater.

2 - Yes, I agree, I believe that is the case. I will check tomorrow with MIBOR staff to confirm, but for now, please proceed as if that data will not be available.

Thanks. Posted by Peskoe Ben(unfuddle username: bpeskoe)

Battler45 commented 4 years ago

George & Vihar,

We received an email today from a staff member at MIBOR / Indy stating that we are displaying listings on the HomeToIndy.com site which should not be displayed.

In the data feed, there is a field, "Public Internet Yes/No". When this field is set to "No", we are not meant to include these listings in our feed and display them in the site.

The staff member cited the listing MLS #21163069 as one such listing for which this field is set to No, but which is displayed on the HomeToIndy.com site.

Can you please check the programming for the Indy Updater and correct this issue, and then re-run a full update of all Indy listings so that these "No" listings are no longer displayed on the site.

This is a priority for us, and it would be good if we could have this done by tomorrow (Wednesday), my time.

Thanks. Posted by Peskoe Ben(unfuddle username: bpeskoe)

Battler45 commented 4 years ago

George,

I agree - it should be "Change Date". If we filter records by that date, the server should return records that have been changed since the specified date. Also, can you pls share the MIBOR meta-data? I'll use it as reference for calrifying / confirming other this and mappings.

Thanks. Posted by Shah Vihar(unfuddle username: vshah)

Battler45 commented 4 years ago

George,

I've tested the photo-migration app. It is working alright. I've made a couple of changes - the main change being to continue processing photos until all photos have been migrated.

As for listing removal - since existing records will also have SourceUniqueID value, the current implementation is fine. No further changes needed to be made. With this I think we've more or less completed the MIBOR updater. Can you pls thoroughly test the updater one last time? Once that is done, I'll test it on the development server.

Thanks Posted by Shah Vihar(unfuddle username: vshah)

Battler45 commented 4 years ago

George,

You are correct. The dev. site also connects to the live database. That is the reason it didn't display any result for MLS search. Ben and Vihang are working on adding new front-end sites, and, therefore, they need to connect the dev. sites (on sierrainteractive-1) to live DB so we would not be able to perform visual comparisons.

Btw, I quickly checked the latest MLS records on the dev DB. Pls find below my comments:

1 -- It looks like the newly added listings are not geocoded. That is, the Geocode_ID value for such listings is 0. To give you a brief overview of geo-coding - we have a table, Geocodes, which stores details like latitude, longitude, address etc... This info. is used in displaying listing on map, performing search by map area etc... So when a new listing is added to the MLS table, we also associate it with a new Geocodes record. If latitude / longitude info. is available from the RETS server, then those values are used in creating a new Geocodes records.

The updater also runs Geocoding step. This step queries 3rd-party service like Yahoo / Google to get missing geocoding details. And, associates listings with new Geocodes records. So for all those listings, which don't have latitude / longitude info. available, the geocoding step populates missing details for them.

Can you pls verify this in the MIBOR updater?

2 -- None of the listings have open-house details. Can you pls check if the open-house step is executed for the MIBOR region? This step should populate info. like open-house date / time etc...

3 -- I'm not sure if had the photo-downloader step enabled for this specific test. If it was, can you pls check why photos have not been downloaded for newly added listings? Also, no records have been added in the MLSPhotos table.

4 -- When you have chance, can you pls add some logging in the MIBOR steps' programming? This will help us in verifying updater steps, location errors (if any).

I'm currently working on finishing one of the open tasks. Hopefully that would be completed in another day or two. I'll then be able to test the updater thoroughly.

Thanks.

Posted by Shah Vihar(unfuddle username: vshah)

Battler45 commented 4 years ago

George,

Per Ben's feedback,

1 -- the current format of Features* value is fine. Pls ignore my comment.

2 -- the approach to remove all photos and re-download them sounds fine for MIBOR region. In SoCal RETS server, we do have access to Media resource, and, therefore, we could make use of the modified photos' details. Unfortunately, we don't have access to Media resource in MIBOR RETS server so pls continue you with your approach of deleting and re-downloading all imagescomment).

3 -- since we have access to only Active / Pending listings no need to make any changes for the same. The MIBOR importer is already populating the Status field (MLS table) with appropriate value from the RETS field, 134 (Status) so no further change needs to be made in this regard.

Pls make changes for the other points discussed above. Pls let me know if you have any queries.

Also, in reviewing the photo-processing code I realized that it is the same for both regions. Therefore, I've renamed one of the PhotoProcessor classes to DefaultPhotoProcessor and removed the other. As with DefaultGeocoding, we can continue using this class for other regions unless there is any change in the approach.

Thanks.

Posted by Shah Vihar(unfuddle username: vshah)

Battler45 commented 4 years ago

Vihar,

Unfortunately I can not remote into 208.106.250.174 to update database with my scripts - maximum number of allowed connection is exceeded. Also I have again question about git - I've run git fetch command but I don't see any changes (there should be changed ImageHelper.cs and Constants.cs). What I have to do more?

Thanks Posted by Molchanov George(unfuddle username: george)

Battler45 commented 4 years ago

Ben, Vihar I have some questions regarding common rules of fields' mapping. 1) What should I do if I can not find an appropriate source field for some field of object Property (for example SoCal model has data for ListAgentLastName - p.AgentLastName = resultSet.GetString("ListAgentLastName"), - but I can not find such field in MIBOR's fields). 2) On the other hand there are a lot of MIBOR fields which are not mapped with object Property. 3) Special case is the composite fields (like FeaturesExterior). Such fields in SoCal implementation have fixed headers for specific source fields (for example in composite field FeaturesExterior can be added field PatioFeatures with header Patio). Are these headers fixed or for MIBOR field FeaturesExterior can contain other data with other headers? And if yes - do you have some ideas which MIBOR fields can be included in composite fields of object Property? Posted by Molchanov George(unfuddle username: george)

Battler45 commented 4 years ago

Ben, Vihar I added code to photos downloading. As I understand ASP system we have not an opportunity to update photos (like in SoCal importer) - we have to delete all photos for property and download again. Is it correct? Thanks P.S. I will work with remote testing tomorrow Posted by Molchanov George(unfuddle username: george)

Battler45 commented 4 years ago

Vihar, What do you mean by MIBOR meta-data? I'm using T5_field_info_RETS__Updated01-17-09.xls document attached in first post. Or you want to see how I perform this mapping (so I can give my code of "Parse.." function). Thanks Posted by Molchanov George(unfuddle username: george)

Battler45 commented 4 years ago

Vihar,

Thank you. I able to connect to 208.106.250.174 now.

Thanks Posted by Molchanov George(unfuddle username: george)

Battler45 commented 4 years ago

Ben, Vihar,

I have some questions regarding outstanding issues (geocodes and open houses details).

  1. Geocodes. It looks like old updater doesn't get latitude and longtitude from MIBOR response. Isn't? I found in meta data some fields which probably can be used for this - DrivingLatitude and DrivingLongitude but I'm not sure. Anyway in my tests all MIBOR records don't return values for these fields. Also I've added functionality of geocode processing (similar to SoCal functionality) and it works.

  2. Open-house updating. Can it be that MIBOR's user doesn't have some rights to get open house data? I've tried to get such data and I received error "User does not have access to Resource named OpenHouse".

Thanks Posted by Molchanov George(unfuddle username: george)

Battler45 commented 4 years ago

George,

We needed to change the data-types of Latitude / Longitude columns (Geocodes table) to numeric(12,9). This led to change in a couple of updater specific store-procedures, and, hence, change in the new updater system. I've made these changes, and, have pushed the same to the updater repository. Pls fetch the latest changes and update your local copy.

Pls note that you'll also need to make a minor change in the way latitude / longitude values are set in the MIBOR import. Until now these values were set as string, but, now they would need to be set as Double. Could you pls update MIBOR import code to set latitude / longitude values as double. You can use the ToDouble string extension I've used in SoCal importer. Pls let me know if you have any queries.

Pls note that I've applied these changes to the dev. sierrainteractive-1 DB. I've attached an SQL script that you can run on your local DB to apply data-type specific changes.

Thanks. Posted by Shah Vihar(unfuddle username: vshah)