Add a "use default" option to Data Manager so that users can get the param layers from web services.

[jdduh]

This feature will be implemented in 2017-18 project period.

[jdduh]

The image services are at: http://basins.geog.pdx.edu/arcgis/rest/services/BAGIS_Databin

Please use the 1981-2010 PRISM web services as the default.

It seems that the tags field in the original dataset is propagated to the tags description of the image service. See http://basins.geog.pdx.edu/arcgis/rest/services/BAGIS_Databin/PRISM_1971_2000_tmax_august/ImageServer/info/iteminfo for an example.

Here is what it looks like in ArcMap.

[lbross]

Does the Data Manager need to support web feature services or just image services? The current implementation supports feature classes in a geodatabase but I don't believe any of the layers in the default data bin are vectors.

[jdduh]

We can limit the webservices to only image services. That means a vector layer needs to be converted into a raster layer first.

[lbross]

After many fits and starts, I have been able to access the IMetadata object and our BAGIS Tag from an image service! Unfortunately the tags in the sample image services aren't in the correct format. The commas need to be semicolons, otherwise the xml parsing fails. Example tag for a temperature layer: BAGIS Tag < Please do not modify: ZUnitCategory|Temperature; ZUnit|Celsius; > End Tag

[jdduh]

If I change the commas to semicolons, will BAGIS-P be able to handle the updated format? Or you can make it to parse either format?

[lbross]

The format that I requested (semicolons) is actually the original format. I copied the tag in my comment from one of the temperature layers in PE_SR_Obs_nmonth_Databin_10072016.gdb.

I think what is happening is that the field we are using (Tags) is expecting a comma-separated collection of words. When you put a comma inside of the BAGIS tag, it parses it apart. BAGIS-P needs it to be a single string to parse.

[lbross]

Any chance you can update the format of the units tags on the server this weekend?

[jdduh]

ArcGIS server automatically replaces semicolors with commas when publishing the services. If I replace the commas with semicolons in the tag, then the BAGIS tag becomes scrambled (see image below). ArcGIS Server treats semicolons as default tag delimiters. Please suggest a workaround so that the tag works for both addins and the server.

[lbross]

Nice. Can you try replacing the commas with pipes? '|'? From the desktop client, the units will be read-only on the services so I only need to update the function that reads the units from the metadata. And I already have to do this to read the metadata from a service. I will read the contents of the tag 2 different ways depending on whether it comes from a service or file.

[lbross]

I'm trying to decide what to do about checking the projection. Currently we have a hard-stop built-in if the datum of the layer to be clipped doesn't match the DEM. This seems overly restrictive, especially when using the image services that we own. I could pop a message like we do when generating an AOI in BAGIS V3 but the data manager is built to clip several layers at a time and this might get annoying. Also thought about putting a warning message on the grid but that could get overwritten if there was an error during the clipping process.

Another alternative is to check the datum when the user selects an AOI, like we do the cell size. We could pop a warning here, like we do with the cell size. They can opt out of adding the AOI. But the actual clipping would run in batch mode.

[jdduh]

It seems that the tag problem is related to the web-based tag editor in ArcGIS Server Manager. The editor removes everything between <> (including the brackets), no matter what the edits are. I tried to hack the service meta data which are stored in C:\arcgisserver\config-store\services\BAGIS_Databin\PRISM_1971_2000_tmax_august.ImageServer\esriinfo\iteminfo.json for the PRISM_1971_2000_tmax_august image service. Can you check if the tag of this image service works? http://basins.geog.pdx.edu/arcgis/rest/services/BAGIS_Databin/PRISM_1971_2000_tmax_august/ImageServer

[lbross]

Again. Nice. Something is still not right with the metadata. My guess is there are still some commas in there that are throwing it off. Also, if you try to look at the metadata using ArcCatalog, it says 'This item's XML contains errors'. You can do this by right clicking on the service in ArcCatalog and selecting Item Description.

Have you tried changing the semicolons to pipes before publishing the image service?

On a related note, I've found that the metadata doesn't isn't transferred to the raster when you clip from an image service. It is when you clip from a file raster (the main reason we decided to go this route) so I will need to write some code that transfers the metadata after clipping to an AOI.

[lbross]

I realized over the weekend that we don't want to change semicolons to pipes because we are already using pipes within the BAGIS tag to delineate between related pieces of information. I suggest using a "!" instead. Here is a sample of the suggested tag: BAGIS Tag < Please do not modify: ZUnitCategory|Temperature! ZUnit|Celsius! > End Tag

The ! will be a terminal tag denoting the end of that piece of information. If AGS doesn't like that, we just need to pick something (not a comma or semicolon) that wouldn't normally be in the tag contents.

[jdduh]

I have to republish the image service to make the updates. The only image service that I updates is:

http://basins.geog.pdx.edu/arcgis/rest/services/BAGIS_Databin/PRISM_1971_2000_tmax_august/ImageServer/info/iteminfo

If this works, then I will update the other image services.

[lbross]

The new tag works! I wonder if the <> characters are causing a problem when you try to update the metadata through the web ui? These are reserved characters in html. Just for grins, try this string for one of the web services through the AGS UI. If it works, it will save you some time:

BAGIS Tag < Please do not modify: ZUnitCategory|Temperature! ZUnit|Celsius! > End Tag

[jdduh]

You are right. The angle brackets were the problem. I have to use the following text to update the tags in ArcGIS Server Admin.

"BAGIS Tag \u003c Please do not modify: ZUnitCategory|Temperature! ZUnit|Celsius! \u003e End Tag"

I think JSON treats angle brackets as special characters.

[lbross]

Excellent! Which of the PRISM layers should be the default for BAGIS-P? 1971 or 1981? Also, still have the outstanding question about how to check the projection for image services

[lbross]

Something to be aware of during implementation: If you change the name of the layers, as you did for the 4 temperature layers, to be different than they were in the original databin they will need to be re-clipped to the AOI. This is particularly true of the jh layers which are name dependent. The name of the layer in the AOI paramdata.gdb is derived from the source layer in the Data Manager. If the name of the source layer in the Data Manager changes, the link is broken.

UPDATE: This is not true. As long as the 'Name' in the Public Data Source Manager is what the model is expecting, there will not be a problem. We built in this abstraction so we aren't reliant on the actual file name.

[lbross]

Wanted to express concern on cell alignment. It seems that the cells clipped from image services align but if I compare it to a raster clipped from a file .gdb, they don't. I am working with an older AOI where the filled_dem came from a file gdb DEM. All of my prism layers and new databin layers align but they don't align with the filled_dem. Not sure this can be fixed but we may need to direct users to go all one way or the other with their AOIs (image service vs file gdb)

[jdduh]

All databin image services are published. See: http://basins.geog.pdx.edu/arcgis/rest/services/BAGIS_Databin Set the default PRISM temperature services to the 1981-2010 data set.

[lbross]

Reviewed the existing clip code for BAGIS-P. We check to see if the layers use the same datum but not the same projection. We appear to be following this protocol in BAGIS-H as well. I can write a function to check the datum of an image service. Is this enough to deal with the projection question we discussed last week? Also a reminder to review my earlier note about cell alignment. Is this going to be a problem?

[lbross]

1. Added a progress indicator when the Public Data Manager loads; With the default image services, it can take a noticeable amount of time
2. Added all remaining image services to yaml configuration file on basins; Used names and descriptions from existing fgdb data bin
3. Tested by clipping entire databin to AOI and re-calculating the parameters. It worked fine.

Posted new v1.9,9-b1 with this functionality to the release page. This enhancement is complete.