Download business logic and layer files on the fly instead of shipping with the AddIn

[philipbaileynar]

Scoping out arcRAVE downloading business logic and symbology from RiverscapesXML instead of being bundled with the software.

Caveats

Experience suggests that ArcGIS does not allow multi-threaded processes. Therefore ArcRAVE will not automate or asynchronously download resources (business logic and symbology.) It will be a synchronous "UI blocking" feature that is triggered by the user. The user will have to wait for the process to finish! No cancel button. Nothing! (For now.)

Features

1. ArcRAVE will still ship with resources (business logic and symbology). But these will be git submodules in the main git repo, linked to the RiverscapesXML repo.
2. The order in which ArcRAVE looks for resources will continue to be:
   1. Adjacent to the project XML file.
   2. AppData folder.
   3. Software deployment
3. A new "Update Resources" menu item will be added to the RAVE toolbar (just like QRAVE). This will download the resources into the users AppData folder. i.e. once this is performed once, it essentially hides (or makes redundant) the resources shipped with the software. The download will lock the UI (for now) with a wait cursor and progress dialog. Try/catch will be used to ensure correct handling of exceptions and prevent locking ArcGIS in the case of an error (such as no internet).

Update Resources workflow

1. Call the git API to get a list of business logic files. @MattReimer what's the URL for this call please?
2. Loop over business logic files placing them in the correct folder.
3. Call the git API to get a list of layer files. @MattReimer are we going to store these non-XML files in RiverscapesXML?

[MattReimer]

1. Call the git API to get a list of business logic files. @MattReimer what's the URL for this call please?

No API url. Eveything's done with a public manifest file you need to download.

2. Loop over business logic files placing them in the correct folder.

Yup. That's basically it. You need to do a presence/absence check and an MD5 checksum on each file too though

3. Call the git API to get a list of layer files. @MattReimer are we going to store these non-XML files in RiverscapesXML?

Again, no API call. Lyr files are already in the manifest so it should be the same process as (1)

---

Let me ellaborate. I'll link to the python code where I can.

The bulk of the code is in net_sync.py and all the actual values are in config.json

Steps:

1. Update the digest.
   1. Download the JSON file .
   2. Parse the JSON and pull what you need out of it (in your case BL xml, .lyr files and the basemaps xml file)
2. Sync Files
   1. Loop over the entries from (1)
      1. IT DOESN'T EXIST: Easy. Download it
      2. IT DOES EXIST: If they exist locally you'll have to do an MD5 hash and compare it to what's in the JSON file to see if you need to replace it.
   2. Loop over any files that aren't in the JSON file and delete them. This allows us to clean up BL files that get removed from the repo

[MattReimer]

FYI MD5 Checking doesn't seem that hard in DotNEt

From: System.Security.Cryptography.MD5

static string CalculateMD5(string filename)
{
    using (var md5 = MD5.Create())
    {
        using (var stream = File.OpenRead(filename))
        {
            var hash = md5.ComputeHash(stream);
            return BitConverter.ToString(hash).Replace("-", "").ToLowerInvariant();
        }
    }
}

[MattReimer]

Couple of notes:

1. Status

It took a while the first time (20+ seconds) and there was no status or progress of any kind. This can be really annoying for a user who might start wondering if Arc has hung. Not saying you need a download progress bar (although that would be nice) but maybe you could just update the ArcMap status bar with some kind of message saying something dumb like "Syncing 2/254" or whatever.

2. First download had everything

After installing the addin it downloaded everything. This may not be a real issue. I'm not sure what was in that temporary addin you sent me. My worry is that the shipped versions of the resources have a different MD5.

3. Resource paths not the same as QGIS

This is VERY minor but potentially confusing for symbology creators

In QGIS:
qrave_toolbar\resources\symbology\Shared
RAVE\Symbology\esri\Shared

Note the esri folder that is redundant.

4. Basemaps Broken

When I add the google imagery I get:

USGS hydrography seems to still work. My guess is that we might not have implemented xyz tiling in ArcRave yet. If so this might warrant its own ticket and we can maybe push this to the next patch if it's hard

<Layer name="Google Imagery" type="xyz" url="https://mt1.google.com/vt/lyrs=s&amp;x={x}&amp;y={y}&amp;z={z}">
<Layer name="Watershed Boundary Dataset" type="wms" url="https://hydro.nationalmap.gov:443/arcgis/services/wbd/MapServer/WmsServer?">

5. VBET not loading

@KellyMWhitehead found this. I looked into it. I can tell that my business logic for VBET is failing to load (so it must be excepting somewhere) but I can't figure out why. The BL XML for VBET does validate for me FYI

[philipbaileynar]

This is implemented:

• Kelly's VBET problem was fixed by changing the way ArcRAVE finds business logic #167
• Broken base maps is being tackled in #169