search

PSU-CSAR / django-snodas

Project to store SNODAS daily rasters and serve as tiles, with some analytical capabilities.
2 stars 0 forks source link

BAGIS-Pro and BAGIS v3 dependencies on ebagis webapp #3

Closed lbross closed 1 year ago

lbross commented 1 year ago

We have agreed to replace the current ebagis ui with a static page for the snodas app. We may have forgotten some dependencies on the ebagis webapp.

  1. The default settings for BAGIS v3 and BAGIS-Pro are served up as JSON files from ebagis. I'm not sure how this works, but the settings files are visible here and here. To update the contents of these files, I update the C:\ebagis\desktop_settings.yaml.
  2. There is an ArcGIS .lyr file that is downloaded from here from the BAGIS v3 settings form. The BAGIS_Reference_Maps.lyr is also stored in the C:\ebagis folder on basins. In BAGIS Pro we have the capability to download files like this from ArcGIS Online, but ArcMap doesn't have this capability.

None of these files is password-secured. @jkeifer: Can you think about how we can continue to support these functions if we EOL the ebagis webapp?

jdduh commented 1 year ago

The second link in 1 doesn't work. How does ArcGIS Pro download the .lyr files? Is it just based on https or a customized django API?

lbross commented 1 year ago

The second link worked for me. It just took a little longer. No web content appears, but the .lyr file is downloaded to your browser downloads folder. ArcGIS Pro downloads .lyr files from AGOL using a special item url. BAGIS V3 downloads the file from Basins. I think there's something going on in the django API given the url. It is not a traditional FTP connection. I'm not sure how @jkeifer implemented either of these functions.

jkeifer commented 1 year ago

The lyr file being a static file could likely be served using the IIS static file handler. But I am not so sure about the desktop settings file; I think we could also use IIS static file handling with an explicit json mime type for the content, except that the file is yaml and it looks like we are only serving a portion of it via ebagis.

Perhaps what we could do is setup a tiny IIS project that has a script to load these files into the proper directory structure to emulate the same file paths, and that script would do the extraction/transformation of the settings from the yaml file into a json file. I'm not sure if we could match the urls exactly, but it might be worth a try.

How often are the settings updated? Would also having to update this "static file caching layer" be a problem in the event the settings or lyr file was updated?

Another option is to extract out this file handling into a separate mini django project, but that would have a higher level of complexity and maintenance burden, so I would prefer not having to go that direction.

lbross commented 1 year ago

We "own" the clients that consume these files so changing the urls isn't a huge deal, especially if it simplifies things and is more consistent going forward. It's a slightly larger issue for the .lyr file because that is used by BAGIS v3 which has been released to NWCC and is no longer regularly updated. But I don't think it's a deal breaker. For the .yaml files, it would be pretty easy for me to rewrite them in JSON, if that makes it easier to serve them as JSON files using a static file handler. These files are consumed by BAGIS-PRO and we are the only ones currently using it so changing the url is fine. I could just make a few code changes. The settings can be updated frequently and that does need to be reflected in the served-up JSON. The .lyr file is pretty static. How hard would it be to set up a web app with .lyr and .json file handers on Basins IIS so we could test this out? MS IIS File Handlers

jkeifer commented 1 year ago

As a PoC, I have (or am attempting to have) lyr file and desktop settings served up by the SNODAS IIS static file handler. For example, https://snodas.geog.pdx.edu/static/desktop_settings.json gets you the yaml converted directly to json format.

However, I noticed that the existing settings endpoint accepted a module name parameter, and was pulling out the settings for a specific module to only return those. So I wrote a little script to not only do the yaml to json conversion, but also split the output into a json file per module, all served under https://snodas.geog.pdx.edu/static/settings/:

The lyr file is a bit more tricky because it doesn't have a known mime type with that extension, and IIS is not currently serving it. I'm still working on that...

jkeifer commented 1 year ago

Oh, turned out setting a custom mime type for the .lyr extension was easy. I added

    <staticContent>
      <mimeMap fileExtension=".lyr" mimeType="application/octet-stream" />
    </staticContent>

to the web.config in the static directory and now you can hit https://snodas.geog.pdx.edu/static/BAGIS_Reference_Maps.lyr and get the file.

jkeifer commented 1 year ago

After having done all this, I do wonder if trying to serve these static files ourselves is in fact the right approach. If we have to change the application references for these files anyway, then we could switch do a different hosting mechanism altogether.

Probably the easiest way I can think of for serving static files is via GitHub: we'd simply need to drop them in a repo, and then they have the added benefit of version control.

jkeifer commented 1 year ago

As a PoC of the served-via-GitHub idea, I created a new repo with these files, which can be accessed as follows:

If changing the clients to use these URLs is not overly problematic (seems like it would be just as problematic as the IIS URLs using the static file handler), I would suggest this is the best path forward as 1) we don't have to own any infrastructure or services here, 2) it gets these files into version control, and 3) these files can be updated from anywhere (no dependency on basins).

lbross commented 1 year ago

Thanks for spending some time on this @jkeifer. The GitHub solution could work. I pointed BAGIS-PRO at the bAGIS Pro settings .json and it picked them up. Is there any way we could store formatted .json in GitHub? Currently it's one long string which makes it hard to read and update. I will check the .lyr file next.

lbross commented 1 year ago

BAGIS v3 is able to download the .lyr file from GitHub.

jkeifer commented 1 year ago

I dumped the json the same as it was being formatted for the API response by ebagis, hence the compact formatting. But formatting it with whitespace is simple and I just did it. Let me know if that causes any issues.

lbross commented 1 year ago

Thanks Jarrett. This looks much better. For some reason the API was formatting for the browser even though it comes across as compact when C# uses it. I've migrated BAGIS-PRO to these configuration files since we are the only ones currently using it. When we get ready to put ebagis to sleep, we'll need to release new ArcMap add-ins for bagis v3 and bagis-p so that they are pointed at the new url. I can do this easily for 10.7 but I'm not sure about a new version for10.5. I think we're keeping 10.5 around because the models used by BAGIS-P don't work on 10.7 but the master code line (10.7) looks like its way out of sync with 10.5.

lbross commented 1 year ago

New urls for the config files in their (hopefully) final repositories:

I thought I might be able to store the .lyr file on AGOL but couldn't find a url that BAGIS v3 could use. New versions of BAGIS-P (BAGIS_P_1.11.2-b1.esriAddIn) and BAGIS v3 (BAGIS_3.5-b1.esriAddIn) are available for download on the basins server. Per Geoffrey's request they have updated version numbers. And I "think" they will work on both 10.5 and 10.7. When I looked at the FTP server it looked like we were posting a single version for both clients. I'm sorry my memory is foggy on this. It's been so long.

I'm going to close this issue and @jkeifer can delete the bagis-conf repo at his leisure.