search

Battler45 / SierraMigrationToGitHub

Migration from unfuddle to GitHub
0 stars 0 forks source link

Convert Region 25 (MIBOR / Indy) to new updater #203

Open Battler45 opened 4 years ago

Battler45 commented 4 years ago

George,

The first project on which you'll be working will be to convert an existing MLS data feed to the new updater system.

Vihar will be able to provide you with the technical details for installing GitHub and downloading a local copy of the current code base.

Vihar, will you also please provide George with whatever server and / or database access he needs for this initial project? Please share with him any necessary logins and create any new users as necessary. Also, I think that converting region 25 will be a good place to start because it shares the same MLS data provider (Marketlinx) as the one you have already converted (SoCal MLS). You can find my existing code in:

C:\Inetpub\mls4.sierrainteractivemls.com\admin\mibor

on DEDQ61

The login credentials to the RETS server are:

  • UserID: 31041rets

  • Password: BENJ1001

  • UserAgent: sierrainteract

  • Password: h1ghplateau

RETS URL: http://inr.rets.interealty.com/Login.asmx/Login

As you'll see when you examine my existing code, I was using the sysid name for the database fields rather than the long names for the fields (which were being used for SoCal MLS). I did this because I had a lot of difficulty with the RETS Connector software that I was using to download the daily data files. To help translate between the two, I've attached a spreadsheet provided to me by MIBOR (which stands for Metropolitan Indianapolis Board of Realtors) that cross-references the long field names with their numeric IDs.

Please let me know if you have any questions.

Thanks.

Posted by Peskoe Ben(unfuddle username: bpeskoe)

Battler45 commented 4 years ago

George,

For the purpose of testing, I think it would be good to set the LastUpdateTimeStamp value to previous date i.e. 30th Aug. This should download sufficient number of records, which you can then compare by reviewing live and development database side-by-side. In addition, as Ben had mentioned earlier, you can also compare the live (www.hometoindy.com) and development (hometoindy.sierrainteractivedev.com) sites to make sure the correctness of data.

As for testing the MIBOR updater on the dev server, my thought is to do this by executing command-line test application. This will allow you to test the MIBOR region updater in isolation.

The development server (75.103.119.144) has a Test App folder (D:\Updater System\Test App). Pls FTP the latest command-line executable and other associated DLLs to this folder. The Sierra.Updater.Tester.exe.config file already contains server specific settings so you won't need to change it unless you want to test the email alert functionality. In that case, pls change the TestEmailAddress param with appropriate value.

Important points:

1 -- If you need to update the config file, pls make sure that LogEmails is set to true (otherwise emails would be sent to actual leads)

2 -- The connection-strings are updated to point to the dev. DB (sql4.sierrainteractivemls.com) so you can remove the SoCal specific updater steps from the Updater_MLSRegionUpdateSteps table in dev DB; the SoCal MLSRegion_ID value is 10. This will make sure that only MIBOR records are downloaded / updated.

3 -- Pls don't FTP the following two files: librets-dotnet.dll and librets-pinvoke.dll; they are of 64-bit version. Uploading the build version (which is 32-bit version) will result in error.

I've set up a new user, georgem (#G30rg3#) on the development server (75.103.119.144). Pls remote into the development server using these credentials. Also, you can use the following FTP settings to FTP console app to the Test App folder directly: 75.103.116.57 sierraupdaterdev #Med2a$123#

--

As for side-by-side database comparison, you can remote into the following server: 208.106.250.174 using the same credentials i.e. georgem (#G30rg3#). I've already opened an SQL Server session, and have connected to the live DB (sql2.sierrainteractivemls.com) and development DB (sql4.sierrainteractivemls.com).

Just in case you need to connect to either of these DBs, the username and password are s13rra and 1nteractiv3 respectively.

Pls let me know if you have any queries / need additional info.

Thanks.

Posted by Shah Vihar(unfuddle username: vshah)

Battler45 commented 4 years ago

Vihar,

Can you explain to me how you are able to delete listings from SoCal? Since the same company provides the data feed service for both SoCal and MIBOR, hopefully we can use the same method. If we do not have access to a needed resource, let me know and I will contact the MIBOR representative and Marketlinx to see if we can get access as needed.

Thanks Posted by Peskoe Ben(unfuddle username: bpeskoe)

Battler45 commented 4 years ago

George,

Re. small application (comment-#71) - I realize that there would thousands of records in the MLSPhotos table for a region so it won't be good idea to select all records in one-go. My thought is that it would better to process photos in batches of x number of records. I'm doing something similar in generating email-alerts. If you can take a look at the EmailAlertService.cs file, the SendAlertsInBatch method retrieves records in batch of 100 records. We can do something similar in moving photos to new folder structure. I think it would be good to process photos in batch of 1000 records.

In addition, it would be good to retrieve only unprocessed records. That is, when a photo is moved to new structure, the corresponding MLSPhotos record's PhotoServer field would be set to 1. So in retrieving batch of MLSPhotos records, we could add a filter to retrieve photos that haven't yet been processed (i.e. having PhotoServer = 0). This will allow us to deal with a situation where the application may have to be restarted / rerun.

Pls let me know if this sounds fine.

Thanks. Posted by Shah Vihar(unfuddle username: vshah)

Battler45 commented 4 years ago

Ben,

Yes, that is the only other approach we can use for removal of old listings.

George - can you pls make changes in the listing removal step for MIBOR accordingly? My thought is that we can make use of the SourceUniqueID value stored in the MLS table for this. The removal step will request for all active listings (no date filter), the only value we would require is the sysid value, which is unique, and, therefore, the query should not be time-consuming. Once we have all the sysid values of the active listings, we can use them to compare against sysid values stored in the SourceUniqueID field. The MLS records with non-matching SourceUniqueID field can be removed (along with their associated records).

If you find the query time-consuming, one possible way to overcome it would be to use Offset value. It is like paging. That is we can request resources from REST server in batches of 50 / 100 records. So, in that case, we can use this to download all active listing sysids in batches.

Pls let me know if you have any queries.

Thanks.

Posted by Shah Vihar(unfuddle username: vshah)

Battler45 commented 4 years ago

Vihar & George,

Thanks, everything appears to have updated correctly and to be working properly now.

George, I do have one more question for the Indy updater -- the contact there says that the Sold data for this region should now be available through the RETS feed. Can you check the RETS metadata for this reason and see if the Sold data is now available as a new class?

Thanks. Posted by Peskoe Ben(unfuddle username: bpeskoe)

Battler45 commented 4 years ago

Ben,

RE: #57 - I had worked on a project that used Amazon S3 service for hosting photos. It is fairly straightforward (from technical viewpoint). And, from what I have read about Amazon CloudFront (CDN), it would be straightforward to use it in conjunction with S3 service.

Even though I don't have experience of using Auzre service, I believe it would be straightforward as well. And, I guess it would be easier to use it in a .NET based project like ours. So it eventually boils down to costing of these two services. Both these services do provide calculators for determining approximate costs

That being said, I think we'll need fairly accurate data to evaluate costing of each service. Would it be possible to get an approximate data for the followings?

My guess is that the monthly cost should be within $75-$100 range (or even less). Please note that this cost includes both storage and CDN charges. I'm not sure about our existing costs. If the cost of ramping up Crystaltech cloud server's hard-disk space to say another 100GB is fairly low and monthly cost of Level-3 CDN service is also lower, perhaps we should continue with the current approach of hosting images on the Crystaltech cloud server.

Pls let me know your thoughts.

Thanks. Posted by Shah Vihar(unfuddle username: vshah)

Battler45 commented 4 years ago

Ben,

Yes, in moving to new updater for MIBOR region, it would need to pick up from where the old updater leaves off. I believe except for the photo processing / storing, rest of the processing would be pretty much unaffected. That is, the new updater system will not delete any existing listings records so the leads' associations with the current listings will remain unchanged.

As for photo processing, I believe it would be good if we can make use of the existing photos; thus eliminating the need to migrate (re-download) photos for existing listings. One possible approach would be to introduce a new field in the MLSPhotos table say, PhotoServer (tinyint). This field will indicate the server where the photo would be stored. The existing code that builds path for listing images already handles two different paths (i.e. for SoCal regions and for the rest of the regions) so we would just need to change that programming to take into account the PhotoServer value.

Per this approach, the PhotoServer value for existing photo-records would be say 1 for region-10 records and 0 for the rest of the records. Any new photos processed through the new updater system will have the PhotoServer value set to 1. So the front-end site programming will use the PhotoServer value to build listing photo url. It currently makes distinction between SoCal region and other regions, which can be changed to make distinction based on the PhotoServer value.

The only issue with the above approach would be to handle removal of photos on old server through the new updater system. When an existing listing record is removed, its associated photos are also removed. When the "Purging" step is executed for MIBOR region (in the new updater system), it may be possible that all or some of the photos of the listings to be purged are present on the old update server. My thought is that this would need to be handled through FTP interface.

We can go one step further and implement programming, which would be flexible enough to handle additional photo servers. I haven't given this much thought but I realize that we would have to think about this sooner as the SoCal images are already occupying about 30GB of hard-disk space on the Crystaltech cloud server, and MIBOR images will only add to this size. I guess it would be possible to increase the size so we can continue with the above mentioned approach.

I also wanted to discuss possibility about hosting photos on 3rd-party services e.g. combination Amazon S3 (for hosting photos) & Amazon CloudFront (for CDN) / Azure Storage & Azure CDN. If this is viable then we could change the new updater system to store photos on such service. This will eliminate the need to ramp-up resources on updater servers. And, in the long run (after we have moved to new updater system completely) all photos would be hosted on such CDN service, which would definitely reduce traffic / load on the servers.

Pls let me know your thoughts. George - pls let me know if you have any suggestions / recommendations for the same.

Thanks.

Posted by Shah Vihar(unfuddle username: vshah)

Battler45 commented 4 years ago

George,

A few points for your info:

1 -- The SoCal and Indianapolis Geocoder classes were same so I've removed one of them, and, have renamed other as DefaultGeocoder class. It will be used in Geocoding listings for both these regions. In future, if any region's Geocoding step diverges, a new class can be implemented for that region; otherwise we can continue to use the DefaultGeocoder class.

2 -- I noticed that the DateListed value (which represents the data when a property was listed, and is used in calculating DaysOnMarket value) was always set to current date. Apparently, the format of the ListDate field in the RETS was different. I've made changes accordingly. It now correctly sets DateListed and DaysOnMarket values.

3 -- For each region, a couple of addition steps need to be executed viz. Statistics Calculations (it calculates stats like average price / median price etc...) and Generating Email Alerts (sending emails to leads who have opted to be notified when new listings matching their criteria are added). These steps are common; no additional programming is required. We'll just need to associate the region with the corresponding step in the Updater_MLSRegionUpdateSteps table. The step-ids are 21 and 20 respectively. I think the email-alert step (20) can be associated once we have thoroughly tested the MIBOR region updater.

4 -- To make testing of a single region easier, I've added a configurable parameter, MLSRegionToTest in the console-application. When its value is set to one of the region-ids (i.e. 25) it will perform updates for that specific region. When set to -1, it will perform updates for all regions. I think this would be useful as when we convert regions to new updater system.

Thanks.

Posted by Shah Vihar(unfuddle username: vshah)

Battler45 commented 4 years ago

Vihar,

I'm working now on the photo-migration tool. I hope the first revision will be ready today.

Thanks Posted by Molchanov George(unfuddle username: george)

Battler45 commented 4 years ago

George,

You are correct. The dev. site also connects to the live database. That is the reason it didn't display any result for MLS search. Ben and Vihang are working on adding new front-end sites, and, therefore, they need to connect the dev. sites (on sierrainteractive-1) to live DB so we would not be able to perform visual comparisons.

Btw, I quickly checked the latest MLS records on the dev DB. Pls find below my comments:

1 -- It looks like the newly added listings are not geocoded. That is, the Geocode_ID value for such listings is 0. To give you a brief overview of geo-coding - we have a table, Geocodes, which stores details like latitude, longitude, address etc... This info. is used in displaying listing on map, performing search by map area etc... So when a new listing is added to the MLS table, we also associate it with a new Geocodes record. If latitude / longitude info. is available from the RETS server, then those values are used in creating a new Geocodes records.

The updater also runs Geocoding step. This step queries 3rd-party service like Yahoo / Google to get missing geocoding details. And, associates listings with new Geocodes records. So for all those listings, which don't have latitude / longitude info. available, the geocoding step populates missing details for them.

Can you pls verify this in the MIBOR updater?

2 -- None of the listings have open-house details. Can you pls check if the open-house step is executed for the MIBOR region? This step should populate info. like open-house date / time etc...

3 -- I'm not sure if had the photo-downloader step enabled for this specific test. If it was, can you pls check why photos have not been downloaded for newly added listings? Also, no records have been added in the MLSPhotos table.

4 -- When you have chance, can you pls add some logging in the MIBOR steps' programming? This will help us in verifying updater steps, location errors (if any).

I'm currently working on finishing one of the open tasks. Hopefully that would be completed in another day or two. I'll then be able to test the updater thoroughly.

Thanks.

Posted by Shah Vihar(unfuddle username: vshah)

Battler45 commented 4 years ago

Vihar,

I've committed code of photo-migration tool. You can find two new projects Sierra.Updater.PhotoMigrating (classes to perform migrating actions) and Sierra.Updater.PhotoMigratingApp (console application).

Now there is implemented simple writer of images into file system (new folders' structure). It can be easily replaced with any other writer (for example with uploader to some cloud system).

I have two questions

  1. What should be behaviour of application when some photo can not be processed? Should we remove such record from database completely or we have to mark it as problem (it means we need one more field in database)?

  2. How I have to commit database changes? My changes are

Thanks Posted by Molchanov George(unfuddle username: george)

Battler45 commented 4 years ago

George,

Sorry for not being specific. I actually meant the files present under the Updater\Sierra.Updater.Database folder. I believe you've already made changes to the "scheama changes.sql" file to associate steps for MIBOR region. Pls add the new field statement to that file.

Thanks. Posted by Shah Vihar(unfuddle username: vshah)

Battler45 commented 4 years ago

Vihar,

Unfortunately I can not remote into 208.106.250.174 to update database with my scripts - maximum number of allowed connection is exceeded. Also I have again question about git - I've run git fetch command but I don't see any changes (there should be changed ImageHelper.cs and Constants.cs). What I have to do more?

Thanks Posted by Molchanov George(unfuddle username: george)

Battler45 commented 4 years ago

Ben, Vihar I added code to photos downloading. As I understand ASP system we have not an opportunity to update photos (like in SoCal importer) - we have to delete all photos for property and download again. Is it correct? Thanks P.S. I will work with remote testing tomorrow Posted by Molchanov George(unfuddle username: george)

Battler45 commented 4 years ago

Ben,

I've made changes to the production and dev DBs to add a new column, PhotoServer (bit) to the MLSPhotos table. When 0, it will indicate photos hosted on the old server, and 1 will indicate photos hosted on the cloud server. Currently its value is set to 1 for all records in the sierrainteractive-4 DB and 0 for all other records.

As mentioned in the above comment, the photo-migration app is working fine so we can now begin working on transitioning the MIBOR updater and its photos to the cloud server. I'll test the MIBOR updater on development server tomorrow; will update you on the same afterwards.

Also, we can now update the programming to build photo-urls to use the PhotoServer value. I think it would be a good idea to create a common function, which can then be called from all pages for displaying photos. My understanding is that the following pages will need to be updated:

Pls let me know your thoughts. I'll also update the new CAA accordingly.

Thanks.

Posted by Shah Vihar(unfuddle username: vshah)

Battler45 commented 4 years ago

Vihar,

Sounds great, thanks.
Posted by Peskoe Ben(unfuddle username: bpeskoe)

Battler45 commented 4 years ago

Vihar,

Yes, I agree that it makes sense to add a new field, photo-server, to be used in conjunction with the MLSPhotos table for situations such as this.

As far as the best way to copy images, I believe the fastest and easiest way would be to use an FTP program to copy the photos manually to a location on the new (cloud) server from which they could be processed by the new updater into their appropriate folder locations. I'm not able to set up an FTP account right now to connect to the appropriate folder on DEDQ61 to allow this file download due to operations currently taking place on that server, but I will do that tomorrow and forward you the FTP credentials.

If you or George could begin writing the application that will move / process these photos once they are downloaded, that would be great.

Thannks. Posted by Peskoe Ben(unfuddle username: bpeskoe)

Battler45 commented 4 years ago

George,

I've logged off from the server. Can you pls try after 5 mins and let me know if you can log into the server?

As for the changes to be applied to your local copy - you'll need to merge the remote changes into your local branch. This can be done by executing the following command git merge origin master. I'm not sure if origin is the name of your remote branch. You can check the same by executing git branch -r command; this will list all remote branch maintained by the git repository.

Alternatively, you can run git pull origin master. This will first fetch remote changes and immediately merge them in your local repository. So in a way it does the job of git fetch and git merge.

Pls let me know if you have any queries.

Thanks Posted by Shah Vihar(unfuddle username: vshah)

Battler45 commented 4 years ago

Vihar,

Would you mind taking a look at the data on the development server, including the photos and geocodes, and testing to make sure it matches the live region 25 data?

If so, then I suppose we don't need to re-download all photos and geocodes for all Indy listings -- could we just use the new updater system to pick up from where the old updater system leaves off -- perhaps we could schedule that for Thursday morning this week if everything looks good?

Thanks

Posted by Peskoe Ben(unfuddle username: bpeskoe)

Battler45 commented 4 years ago

George & Vihar,

RE: #69, I've contacted MIBOR representatives to see if they can give us access to the requested resources. I will let you know.

Thanks. Posted by Peskoe Ben(unfuddle username: bpeskoe)

Battler45 commented 4 years ago

Ben,

I've made changes in the new CAA and new updater systems to use the PhotoServer value in building photo urls (not published yet). I'm now working on making similar changes in the front-end sites programming. I'm looking into updating the dev. version of search-tools / content pages for the same. I'll update the live sites tomorrow (during your overnight period).

Thanks. Posted by Shah Vihar(unfuddle username: vshah)

Battler45 commented 4 years ago

Ben,

I've checked MIBOR's RETS metadata and below is list of available classes:

Also I found field "Sold Trigger" which probably can be used in data query. But I don't know if it is correct and how to use it. In data feed this field is unavailable and I don't know which values are acceptable. I've tried to use Y, N, 1 and 0 and everytime I got the same set of listings.

Thanks

Posted by Molchanov George(unfuddle username: george)

Battler45 commented 4 years ago

Vihar,

I thought scripts under Updater\Sierra.Updater.Database folder are intended only for sierrainteractive-updater database. Isn't? And my script has to be run on sierrainteractive-n databases.

Also I attach script here for your testing while it is not committed.

Thanks Posted by Molchanov George(unfuddle username: george)

Battler45 commented 4 years ago

George,

I'll merge those changes to the master branch later. There may be some conflict in merging those changes so I'll work on it after finishing my current task. I believe your current work shouldn't be affected by the merge. Pls continue to work / push changes to the master branch.

Thanks. Posted by Shah Vihar(unfuddle username: vshah)

Battler45 commented 4 years ago

Vihar,

I've fixed issues with ListType and SourceUniqueID.

Thanks Posted by Molchanov George(unfuddle username: george)

Battler45 commented 4 years ago

Ben,

Re. #68 - old and new updaters call the same stored-procedure, spUpdateMLSPhoto for updating a photo-record. So perhaps that sp can be updated to set changed-date in the DateAdded field. Pls let me know if this sound fine. I'll update the sp.

Thanks. Posted by Shah Vihar(unfuddle username: vshah)

Battler45 commented 4 years ago

Ben,

I've tested the MIBOR updater on the development server. It worked fine. I guess we can now transition the MIBOR updater to the new updater. However, we'll first need to migrate existing photos to the cloud server. Pls let me know your inputs on comment#93.

Thanks Posted by Shah Vihar(unfuddle username: vshah)

Battler45 commented 4 years ago

George,

As you have time can you pls work on building a small application for migrating photos from old-updater to new-updater? As discussed above, the old-updater server's performance is not always good and so when a region is converted to the new updater system, photos of that region should be moved to the Crystaltech cloud server.

As far as photo processing is concerned, the main difference between two updater systems is the way photos are organized. As you already know, each photo is resized in three different sizes i.e. thumbnail , regular and large. These photos are organized under the mlsthumbs, mlspics and mlslarge folders respectively. The actual organization however varies for some regions. Some of the recently added regions support organization in sub-folders. As of now the following three regions support organization by sub-folders - 10 (SoCal), 24 (Denver) and 25 (MIBOR); for rest of the regions photos are organized under single folder.

To give you an idea about the folder organization on the existing photo serve:

Root-Photo-Folder

    3
      mlsthumbs
      mlspics
      mlslarge

    ...

    25_1
      mlsthumbs
      mlspics
      mlslarge

     ...

    25_100
      mlsthumbs
      mlspics
      mlslarge
...

This structure has been tweaked for new updater system as follows

Root-Photo-Folder

  10
    mlsthumbs
        AA
        BB

        ...

    mlspics
        AA
        BB

        ...

    mlslarge
        AA
        BB

        ...

Going forward all regions will need to conform to the new photo-organization approach (as implemented in the new updater system). This will be done on region-by-region basis as and when regions are converted to the new updater system.

So the goal of the new application would be to organize photos per the new approach. It will have to deal with two types of existing organization

For start, you can implement the application to process MIBOR photos and move them per new organization. It can later be refined to support all regions.

As and when photos are moved to new organization, we would have to set certain flag in the corresponding MLSPhotos record so that the front-end site builds correct photo-url. As discussed in the comment-#52, a new field PhotoServer would need to be added to the MLSPhotos table. When a photo is organized per new approach, the corresponding PhotoServer value would be set to 1 so that the front-end site builds appropriate url.

My thought is that the application can be implemented to perform the following steps

1 -- to read photo records from the MLSPhotos table for MIBOR region (25).

2 -- it will then copy the corresponding photos (thumbnail, regular and large) from the old folder-structure and move to new structure. Pls use the logic of generating sub-folder name from the PhotoDownloader code (GetHashedDirectoryName function).

3 -- if successful, the corresponding PhotoServer flag should be set to 1.

Since the goal of the application is to support all regions, a configurable parameter should be added to indicate the region to process. A couple of additional parameters can be used for setting source / destination folders. Also, it would be fine to create a new console application (similar to the updater console application) under the updater solution so that it can utilize the existing framework / source-code where possible.

Pls let me know if you have any queries / suggestions.

Thanks.

Posted by Shah Vihar(unfuddle username: vshah)

Battler45 commented 4 years ago

George,

My apologies for the delay in responding you. I've verified the changes for ListType and SourceUniqueID; they are working fine.

Can you pls let me know if you have begun working on the photo-migration application?

Thanks. Posted by Shah Vihar(unfuddle username: vshah)

Battler45 commented 4 years ago

Vihar,

Yes, I believe the issue is caused by several new sites having been added since the development db was last updated. Because adding new sites involves addition of records into multiple tables, it might be best to restore sierrainteractive-1 on the development DB. However, would this impact the work that George is doing and / or has already done? (Erasing test data that he has imported.)

I have created a backup of sierrainteractive-1 called sierrainteractive-1_2011-08-29.bak and downloaded that onto DEDN173 in C:\DB Backups. Please let me know if you have any trouble with that.

Your approach for assisting George to test the updater for region 25 sounds good.

Thanks. Posted by Peskoe Ben(unfuddle username: bpeskoe)

Battler45 commented 4 years ago

George,

Just checking in to see how things are coming -- were my last comments helpful, and have you been able to make further progress?

Thanks. Posted by Peskoe Ben(unfuddle username: bpeskoe)

Battler45 commented 4 years ago

Vihar,

I reviewed the changes, all is fine

Thanks Posted by Molchanov George(unfuddle username: george)

Battler45 commented 4 years ago

George,

I've modified the MIBOR updater to ignore listings with "Public Internet" set to "No". I've pushed those changes and planning to run a full update of all Indy listings in the next 15 minutes. If you happen to check this before that can you pls confirm the changes?

Ben - removing the existing listings with the "Public Internet" set to "No" would be a bit tricky in that those listings are already added to the MLS table, and there is no other way to remove them other than removing those MLS records through the backend. I was thinking that it would be fine to execute the spRemoveListingByMLSNumber stored-procedure manually for all such listings. That is, after the full update is completed today, the DateLastUpdate date of all active listings having "Public Internet" set "Yes" would have changed. The DateLastUpdate field of the other listings, most likely having "Public Internet" set "No", would remain unchanged and so it would be fine to remove such listings by executing the spRemoveListingByMLSNumber stored-procedure manually. Pls let me know if you see any issue with this.

Thanks.

Posted by Shah Vihar(unfuddle username: vshah)

Battler45 commented 4 years ago

Please place further comments into Basecamp, here:

https://basecamp.com/1767592/projects/353414-074-convert-region

Battler45 commented 4 years ago

George,

I've merged changes to the master branch and have pushed the same. Pls note that I had created a persistence method, GetAllSourceUniqueIds for SoCal updater. A similar method, GetAllSourceUniqueID was being used in the MIBOR / GLAR updaters. I've removed the GetAllSourceUniqueID method and have changed references appropriately.

Thanks. Posted by Shah Vihar(unfuddle username: vshah)

Battler45 commented 4 years ago

George,

I agree - it should be "Change Date". If we filter records by that date, the server should return records that have been changed since the specified date. Also, can you pls share the MIBOR meta-data? I'll use it as reference for calrifying / confirming other this and mappings.

Thanks. Posted by Shah Vihar(unfuddle username: vshah)

Battler45 commented 4 years ago

Ben,

Can you pls confirm if SchulerBauer site is served from the DEDN173 server (along with hometoindy and joehaydenrealtors sites)? Actually I wanted to verify the changes I've made for building photo-urls. The schulerbauer site doesn't appear to be affected by those changes.

Thanks. Posted by Shah Vihar(unfuddle username: vshah)

Battler45 commented 4 years ago

Vihar,

I implemented the new approach to delete listings of MIBOR region. It works pretty fast (about 28 thousand records from MIBOR RETS server). All is committed excepting sql script with stored procedure for sierrainteractive-n databases (it is attached).

Thanks Posted by Molchanov George(unfuddle username: george)

Battler45 commented 4 years ago

Vihang & George,

I am currently running the old MIBOR / Region 25 updater and it should be finished by the time you are ready to begin working (it should be done by 1am EST). In order to keep from having to change the stored procedure, I've populated the SysID value from the MIBOR region into the ListingHashValue field of the MLS table in our database. It is now configured to update this every day when the rest of the listing data is updated, so you should be able to use this in creating the script to remove old listings as we transition over to the new updater system.

Please let me know if you run into any problems with that.

Thanks. Posted by Peskoe Ben(unfuddle username: bpeskoe)

Battler45 commented 4 years ago

Ben,

Was about to update you on this!! There were some issues in re-running full updates of MIBOR records. It failed thrice with the "Error occurred in logging into inr.rets.interrealty.com server" message. I'm not sure if this had anything to do with the full updates. Moreover, doing a full-update would have consumed a lot of time (approx. 8-10 hours) therefore I aborted the full update and ended up choosing a different approach.

I've modified the Indianapolis listing removal functionality to remove all listings that don't have "Public Internet" set to "Yes". This will not require manual removal of listings. In addition, it will also remove photo files from the server. The modified removal-functionality appears to have removed all listings having "Public Internet" set to "No". Pls check the same and let me know in case of any issues.

As for the point about not displaying address - I verified that we've programming in place to change the address to "000 Confidential Ave." for listings where the "Address yes/no" field is set to "No". There are total 137 listings in the database having this address.

Thanks. Posted by Shah Vihar(unfuddle username: vshah)

Battler45 commented 4 years ago

Vihar,

I see your changes are committed into production branch. Do you plan do commit these into master too? (I'm working with master branch, is it correct?)

Thanks Posted by Molchanov George(unfuddle username: george)