Data loss during RT load process

[SteveGrise]

It looks to me like most of the RT lines are missing on mapsdev cadsde.

From a quick look what I see is that something happened last week on Thursday before the runSynNavigation.py was run at around 7pm.

Earlier the runSyncSites.py process that finished at 4:01pm had the runRoutingExtract.py script embedded. In that RT_Extract.log file only 425 rows were read from line interior:

This makes me think that something happened to the data in between the full reload that finished at 2:17pm and this second time the RT process was run an hour or so later. Oddly there was an RT load process attempted every minute or so in between the larger loads...

The only other clue I see is that the 1200RT-RF drawing had no data in it, but the sync did not run for that drawing so I'm at a loss for how that could be a problem.

I do see that the process was running multiple times on Friday afternoon, often with 0 drawings processed. I am concerned if there is more than one of these processes running at the same time that there is an issue in the sync process that causes data loss.

[SteveGrise]

@aaronch I did notice one issue in the gseRoutingLoader.fmw file. There is a parameter for which database to write to, and inside the FME workbench that was set to the cadsde database not Staging. This is always passed as a parameter to point to staging from scripts, but I suppose there is a chance of something not working correctly at certain points that would cause a truncation of cadsde instead of staging... but that is still just a guess and not likely.

[SteveGrise]

Ran an update to staging then appended to cadsde database. Results:

[SteveGrise]

@aaronch looks like a good result now in Routing.gdb.

[SteveGrise]

@aaronch steps to run reload to staging:

• Open ETL\fme\gseRoutingLoader.fmw
• Run from within workbench (will take a few hours)
• After complete, verify data in Staging database
• in ETL\gseLocal.tbx there is a model that will replace data from Staging to Cadsde, you can use that or another approach since it's just a deleterows/append model.
• result should be correct data in Cadsde database.

[SteveGrise]

Data issue reported on June 28

• appears that TransitionLines are missing except for several drawings from last RT load.
• no clues in processing results
• Transitions only for later part of load - several drawings ~1008 completed and sync'd then 1041_01 the first drawing with transitions - seemed to work after that point...