BUG-000094248 When using an ArcGIS Online polygon feature service as a private layer with stored credentials, and viewing in the Crowdsource Reporter application, the selection polygon is a much different geometry from the original polygon.

[allisonmuise]

Configuration:
ArcGIS Online 3.10

Data:
Feature Service CharlotteLargeExtent (item URL: http://ess.maps.arcgis.com/home/item.html?id=047185ce5f954c5f84ce6fa030cda764, service URL: http://services.arcgis.com/Wl7Y1m92PbjtJs5n/arcgis/rest/services/charlottelargeextent/FeatureServer/0, shared with ESS organization)

Steps to Reproduce:
1. Add the CharlotteLargeExtent feature service to a new ArcGIS Online webmap. Save the map.

2. Copy the CharlotteLargeExtent service URL. In My Content, add an item from the web, and add this feature service as an ArcGIS Server web service. It will be a secure service. Store your username and password with the item and give it a new name.

3. Add this layer with stored credentials to a new webmap and save the map. In the webmap's details, notice that the service URL now starts with utility.arcgis.com.

4. Create a new group, and share the webmaps and both service layers (secure and unsecure) to this new group.

5. In My Content, create a new Crowdsource Reporter application. In the configuration, set the group created above to be the group used for the application. Save the configuration.

6. Open the application, and you'll see two maps - the one with the secured service and the one without. Select the one with the unsecured service.

7. Click on one of the features with the unsecured service. You should see that the selection polygon is fairly close to the actual polygon.

8. Go back to select the map with the secured service. Click on the same feature. You should now see that the selection polygon does not match the actual polygon.

Testing notes:
It appears this difference is due to the tolerance in the quantization parameter, which can be seen in Fiddler. The unsecured service's tolerance is much lower than a secured service's tolerance.
Strangely enough, the quantization parameter tolerance for the secured service matches the resolution for a scale level in the webmap's basemap. For example, Fiddler showed the secured service's QP tolerance to be 76.43702828507324, which matches the resolution for Level 11 of the World Topo basemap (http://services.arcgisonline.com/ArcGIS/rest/services/World_Topo_Map/MapServer).
Even when using secured services with other basemaps, the QP tolerance always matched the resolution for one of the levels of the basemap. The unsecured service QP tolerance does not match the basemap resolution.
It appears that a larger initial extent for the feature service creates a larger quantization parameter tolerance.
This cannot be reproduced in the webmap.
This has been reproduced with various spatial references.

Note: Securing a service is listed on the official Crowdsource Reporter documentation: http://solutions.arcgis.com/local-government/help/crowdsource-reporter/get-started/limit_access_to_public_agol_layers/

Salesforce ID: BUG-000094248
Salesforce Submitter: Mary Long
Salesforce Submit Date: 2/18/2016 4:51 AM
Salesforce Bug Type: Usability
Salesforce Severity: Medium
Repro Data: (n/a)
Work Around: Workaround 1: Use a service without stored credentials. Workaround 2: When publishing the feature service, use a smaller extent.
Product: ArcGIS Online
Functional Category: ArcGIS Online Application Templates
Client Platform: Windows 7.0 64 Bit
Version Found: 3.10
Planned Version Fixed: (n/a)
Comment: (n/a)

[allisonmuise]

arcgis-portal-app-templates #1103

[CTLocalGovTeam]

@allisonmuise,

We are able to reproduce this. However, we are not able to modify the tolerance in quantization parameters while executing the query. We tried to set the tolerance to 0, 1, 5, 10. But this seems to have no effect in the query, the API overwrites the tolerance with the one it generates internally. We tried to supply maxAllowableOffset, but even that is disregarded in the query.

[MikeTschudi]

Observation: Using the test app http://arcgis4localgov2.maps.arcgis.com/apps/CrowdsourceReporter/index.html?appid=a1fdf7c56ab94113adc49c75802f5e7a#, I intercepted and modified the query and got the same results every time. The API is generating a tolerance of 78271.51696399994; I tried 78.27151696399994, 78, 10, and removing the quantizationParameters. The numbers for coordinates, scaling, and translation varied with the different tolerances, but the display was the same in every case:

[MikeTschudi]

Updated observation: There are two fetches for the map features, and both are affected by bad tolerance values.

1. The first one, represented by the light green box above, is fetched when the map is loaded. If the quantizationParameters' tolerance is adjusted for this fetch--or the quantizationParameters parameter deleted--the feature is loaded correctly for general map display.
2. The second one, represented by the aqua highlight triangle, is fetched when a particular feature is selected by name from the list of issues or the light green box is clicked and is used to highlight the map feature.

With the previous observation, I was only intercepting and adjusting the second query. When the API receives the result, it looks to see if it already has that feature layer and feature loaded; if found, it uses the previously loaded coordinates. So even though the service was returning good coordinates as a result of adjusting the quantizationParameters' tolerance, it didn't matter because the API discarded them. In order to get good results, quantizationParameters has to be right for the first query. With quantization removed for both queries, one gets both the map and highlight versions of the feature correctly:

The quantizationParameters' tolerance is set by the API to correct values, but I'm seeing a problem when the setting is occurring. It appears as if there is a different copy of the feature layer for the two queries, and only one is being updated as the map is zoom or resized. In addition, there may be a timing problem where the map is zoomed as a result of a selection, but the tolerance is based on the pre-zoom resolution.

I don't see a reason for secure vs. nonsecure causing the problem yet, but one of the requirements for the quantization parameter to be used by the API is that the service has the parameter "supportsCoordinatesQuantization" set to true. The 4.0 API has the comment, "Only works with ArcGIS Online hosted services.", which may be a clue for the 3.17 API behavior that the app is seeing, too.

[MikeTschudi]

There are three problems of coordination between the app and the JSAPI:

1. There are multiple copies of each map's FeatureLayers in the app:
   • One copy ("A") is created when one logs in
   • Another copy ("B") is created each time that one selects a map from the map select panel
2. Features are queried at one zoom level and tolerance (the rounding of coordinates to the nearest pixel so that coordinate changes too small to be visible at the current scale are omitted) and never updated as the zoom level changes.
3. Tolerance is calculated after a query is submitted that needs that tolerance (and for the wrong feature layer).

An example to illustrate these:

1. Log in as guest
   • Copy A is created and its internal tolerance value is set based on (I suspect) the scale factor and browser window when it was saved. For the BUG-000094248 map, its resolution is approximately 78272 based on a display width of 31308606m/400px.
   • The first map in the app's group appears along with the map select panel.
2. Select the BUG-000094248 map from the map select panel
   • Map BUG-000094248 appears along with the feature selection panel.
   • Copy B is created and its internal tolerance value is set based on the current scale factor and browser window, e.g., 39136 based on a display width of 29743176m/760px.
   • Copy B's features are queried to get the base graphics using the calculated tolerance.
3. Zoom into Nova Scotia
   • Copy B's internal tolerance value is set based on the updated scale factor, e.g., 2446, with each zoom change. Note that only the tolerance is changed--the graphics continue to have the resolution of the original query from step 2 and the Halifax feature appears as a green square instead of a five-sided polygon.
4. Select the Halifax feature from the list
   • Copy A's features are queried to get the highlight graphic using its never-changed tolerance of 78272.
   • Map BUG-000094248 is zoomed and the report details panel appears.
   • Copy B's internal tolerance value is set based on the updated scale factor, e.g., 153.
   • The highlighted outline of the Halifax feature is a triangle instead of a five-sided polygon because it was queried using the tolerance of 78272.
5. Zoom out
   • Copy B's internal tolerance value is set based on the updated scale factor, e.g., 1223.
6. Click on Halifax' symbol in the map (the green square)
   • Copy A's features are queried to get the highlight graphic using its never-changed tolerance of 78272.
   • Map BUG-000094248 is zoomed.
   • Copy B's internal tolerance value is set based on the updated scale factor, e.g., 153.

The two tolerances consistently yield a square and a triangle, respectively, because the graphics are never queried using updated tolerances:

1. A green rectangle: Copy B's feature from the step 2 query using, e.g., a tolerance of 39136.
2. An aqua triangle: Copy A's feature from the step 3 or 5 query using, e.g., a tolerance of 78272.

[allisonmuise]

@CTLocalGovTeam : It appears that this is an issue with Reporter. Could you please review Mike's information above?

[CTLocalGovTeam]

@MikeTschudi , @allisonmuise

Mike, thanks for your analysis. Just to add more information to the analysis, the app actually works off of a graphic layer. When a map is selected, the featurelayer is initialized, but it is turned off. A separate graphic layer is created and maintained to query and display features from the featurelayer. This is done to support displaying features with increasing proximity (1 mile, 2 miles, so on) around the user's geolocation.

We are investigating this further based on the inputs you provided above.

[MikeTschudi]

@CTLocalGovTeam: Thanks. Some background: I put a pair of probes into the FeatureLayer API code (1. where the tolerance is set and 2. where the feature query is made) that tested and reported FeatureLayer instances, and that's how I saw multiple FeatureLayers being created and queried.

[CTLocalGovTeam]

@MikeTschudi @allisonmuise

Reporter app has two instances of feature layer,

1. Copy ("A") is created on application load to create web map list panel
2. Copy ("B") is created each time when user selects a map from the web map list panel

Copy (“B”) is hidden on the map and it is used to query all the features and to show them on the graphic layer.

We analyzed the issue and found following problems:

1. While getting features on load, the extent of map is set to world extent and therefore the features which are at parcel level or city level get generalized as quantization parameters are applied. Due to this the features shown on graphics layer are not same as stored in feature layer.
2. While highlighting the feature from issue-list, we were querying the feature from the layer and here as you correctly pointed out, we were passing the wrong layer instance. (we were using the layer that was created on app load(A) to create web-map list instead of using the current layer(B) from map).
   But still when we updated the code to use the correct layer(Copy "B") instance from map we were seeing the same issue because quantization parameters were getting applied as mentioned above.
   1. The features are queried on load and added to graphics layer, and they are not queried and updated every time the map zooms or pans.

To fix this we did following code changes:

1. Set the max allowable offset to the layer to a very small value, e.g., “0.1” before querying the features. Due to this, quantization uses the offset as the tolerance, when present and the features are not generalized due to the small tolerance.
2. While highlighting used the correct layer(Copy "B") instance from map.

The code change required to fix this is very small ~ 3-4 lines.

[MikeTschudi]

@CTLocalGovTeam Nicely done to figure out a pinpoint fix!

[allisonmuise]

@MikeTschudi @CTLocalGovTeam : Very nice! Thanks!

Is there any expected impact to performance when pulling in all features without generalization?

[CTLocalGovTeam]

@allisonmuise We dont see any change or impact to the performance. The app's performance will remain exactly the same. The features were never generalized prior to this and are fetched on demand (when user clicks load more).

[CTLocalGovTeam]

@allisonmuise, @MikeTschudi

We discovered during testing that the approach of setting the maxAllowableOffset to a small value (0.1) is not successful in all scenarios. It works only when the default extent of the webmap is set to a large scale , e.g., between 20k-50k approx. When you set the default extent to a large scale, e.g., >100k - 1million (city/region), we still see that features either are incorrect or do not appear at all.

We can confirm that quantization uses the tolerance from maxAllowableOffset i.e., "0.1" in the queries, we see that geometries are returned for all features, but the result features are not correctly de-quantized for display by the API for reasons we don't know yet.

Another aspect to this issue is that we haven't been able to figure out why this only occurs for secured hosted services with saved credentials. Public hosted services already work correctly without this maxAllowableOffset fix. The API automatically uses a very small quantization tolerance of approx 1.0 for public hosted services and features are rendered correctly, regardless of the default extent of the webmap. We are seeing that the API generates a different set of quantization tolerance parameters depending on whether a service is hosted public or hosted secured with stored creds.

Please suggest if we can have a separate call with Mike to demonstrate our investigation and findings so far.

[MikeTschudi]

@CTLocalGovTeam Were you able to rule out a correlation between the feature layer parameter "supportsCoordinatesQuantization" and the different quantization tolerance parameters for the services? In the API source, that parameter seemed to be the main reason for the difference, but I wasn't sure.

[allisonmuise]

Notes from the call this morning: Still can't isolate the issue to just API or just app.

@MikeTschudi is going to build a sample app that reproduces the following issues:

• secure layers behaving differently than non-secure layers
• post-zoom quantization parameter value being calculated from pre-zoom extents.

We will approach API team with this info to see what bugs/enh can be logged for a future API release.

[MikeTschudi]

As mentioned in the first note of this issue, the "utility.arcgis.com" change is significant. For Crowdsource Reporter, an unshared secured service URL with stored credentials published to http://services.arcgis.com/ works fine; the same unshared secured service URL with stored credentials published to http://utility.arcgis.com/usrsvcs/servers/ fails.

1. Published multiple times from ArcMap into a hosted service as private, org-only, and public.
2. Took the URL from each hosted service and added it to the org as described in step 2 of the first note. All of the org-only and all but one of the private services added this way were created as services on http://utility.arcgis.com/usrsvcs/servers/ and these all fail: when one zooms into the map, the small graphics are quantized into a line or even out of existence. One of the private services was created as a service on http://services.arcgis.com/ and it works correctly. (I don't know why it didn't go to utility.arcgis.com.)

Compared the JSON service description for the services.arcgis.com private service with the utility.arcgis.com private service; they're identical.

The private service failing in Crowdsource Reporter works fine in Basic Viewer, including with searches.

[MikeTschudi]

1. Published from ArcMap into a hosted unshared service
2. Used its URL to create a public stored-credential service on http://utility.arcgis.com/usrsvcs/servers/
3. Used the stored-credential service in a public webmap

The layer contains 7 jagged-edge polygons drawn at the 1:24,000 scale or larger; with the roughness of the boundary and the large scale, these polygons change radically or disappear with coarse quantization. Each layer also contains 7 triangles that serve as pointers to the small polygons. The polygons are scattered around the Earth.

To test display and query, I created a webpage that loads this webmap; it comes up at 1:150,000,000 or so. While at that small scale, the app uses the operational layer in the webmap to query for all features and add them to the map's graphics layer. When I zoom in to the 1:24,000 level, I see that all polygons are drawn correctly, e.g., here are the two polygons in one location at 1:5,000:

where

• op layer: blue fill
• query: vertical lines

To show wide extent of map combined with tiny polygons, here are all of the polygons:

The API's createMap function is the place where two FeatureLayers appear to be created for each operational layer. No FeatureLayers appear to be created by the API's query routines. Interestingly, none of the queries to the servers used quantization.

These details show the same webmap with bad quantization in Crowdsource Reporter and no quantization in the test program:

Zooming in to the detail in New York:

The complete webpage:

<!DOCTYPE html>
<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
    <!-- based on http://developers.arcgis.com/javascript/sandbox/sandbox.html?sample=map_simple -->
    <meta name="viewport" content="initial-scale=1, maximum-scale=1,user-scalable=no"/>
    <title>Simple Map</title>
    <link rel="stylesheet" href="//jsdev.arcgis.com/3.18/dijit/themes/claro/claro.css" />
    <link rel="stylesheet" href="//jsdev.arcgis.com/3.18/esri/css/esri.css" />
    <style>
      html, body, #map {
        height: 100%;
        width: 100%;
        margin: 0;
        padding: 0;
      }
    </style>
    <script type="text/javascript" src="//jsdev.arcgis.com/3.18"></script>
    <script>
    require([
        "dojo/_base/array",
        "dojo/Deferred",
        "dojo/DeferredList",
        "dojo/json",
        "esri/arcgis/utils",
        "esri/Color",
        "esri/graphic",
        "esri/dijit/Scalebar",
        "esri/symbols/SimpleFillSymbol",
        "esri/symbols/SimpleLineSymbol",
        "esri/tasks/query",
        "esri/tasks/QueryTask",
        "dojo/domReady!"
    ], function (array, Deferred, DeferredList, JSON, arcgisUtils, Color, Graphic, Scalebar,
        SimpleFillSymbol, SimpleLineSymbol, Query, QueryTask) {
            arcgisUtils.createMap("637ccc9591c6472abb4876ba8b3693a1", "map", {
                usePopupManager: false
            }).then(function (response) {
                var map, opLayers, requestArray = [], deferreds, symbols = [], iSymbol = 0;

                map = response.map;
                new Scalebar({
                    map: map,
                    attachTo: "bottom-center",
                    scalebarStyle: "line",
                    scalebarUnit: "dual"
                });

                opLayers = response.itemInfo.itemData.operationalLayers;

                array.forEach(opLayers, function (opLayer) {
                    requestArray.push(_queryOnLayer(opLayer));
                });

                symbols.push(new SimpleFillSymbol(SimpleFillSymbol.STYLE_VERTICAL,
                    new SimpleLineSymbol(SimpleLineSymbol.STYLE_DASH, new Color([0, 0, 255]), 1),
                    new Color([0, 0, 255])));

                deferreds = new DeferredList(requestArray);
                deferreds.then(function (deferredsResults) {
                    array.forEach(deferredsResults, function (deferredResult) {
                        if (deferredResult[0]) {
                            console.warn(deferredResult[1].title + " OK with " +
                                deferredResult[1].queryResults.features.length + " features (" +
                                symbols[iSymbol].style + " lines)");
                            array.forEach(deferredResult[1].queryResults.features, function (graphicBase) {
                                map.graphics.add(new Graphic(graphicBase.geometry, symbols[iSymbol],
                                    graphicBase.attributes));
                            });
                        } else {
                            console.warn(deferredResult[1].title + " failed with " +
                                JSON.stringify(deferredResult[1].queryResults));
                        }
                    });
                });

                function _queryOnLayer(opLayer) {
                    var queryTask, deferred, parameters, queryString, dateobj = new Date().getTime().toString();
                    deferred = new Deferred();

                    parameters = new Query();
                    queryString = "1=1 AND " + dateobj + "=" + dateobj;

                    // add layer definition in query parameters if it is available in layer object
                    if (opLayer.layerObject.defaultDefinitionExpression) {
                        queryString += " AND " + opLayer.layerObject.defaultDefinitionExpression;
                    }

                    parameters.where = queryString;
                    parameters.outFields = ["*"];
                    parameters.returnGeometry = true;

                    queryTask = new QueryTask(opLayer.url);
                    queryTask.execute(parameters, function (response) {
                        deferred.resolve({
                            title: opLayer.title,
                            queryResults: response
                        });
                    }, function (err) {
                        deferred.reject({
                            title: opLayer.title,
                            queryResults: err
                        });
                    });
                    return deferred;
                }
            });
        });
    </script>
  </head>

  <body>
    <div id="map"></div>
  </body>
</html>

[MikeTschudi]

Update: created webmap with three public layers defined using public, org, and private services. Same program as previous note, but with additional symbols for each operational layer. The layers defined using org and private services have stored credentials.

• public
  • op layer: black fill at 50% transparency
  • query result: diagonal
• org
  • op layer: blue fill at 50% transparency
  • query result: horizontal
• private
  • op layer: red fill at 50% transparency
  • query result: vertical

All layers draw the same, and correctly.

[MikeTschudi]

If, instead of calling queryFeatures() on a layer, one calls an instance of QueryTask, this problem goes away in this app for all layer types. But adding queryFeatures() to the test program works. So I think that we have a fix for this particular issue but there's an underlying question that I haven't yet resolved to API and/or app.

@allisonmuise @CTLocalGovTeam, would you please see if these changes work for your examples, too?

[CTLocalGovTeam]

@MikeTschudi @allisonmuise We did investigate using QueryTask in place of QueryFeatures(), however this switch will have a larger impact on the code base. After making the switch, we'd have to explicitly handle layer filters applied in webmap, popup definitions, etc. Because of that we didn't pursue this further.

[MikeTschudi]

@allisonmuise, the sample app is demonstrating that the JSAPI

• handles secure layers the same as non-secure layers
• handles queries at small scale for display at large scales

There's no bug in or enhancement for the JSAPI at this point; we have to keep investigating the app.

[CTLocalGovTeam]

@MikeTschudi Thanks, we'll keep investigating the app further to isolate the root cause. Can you please let us know if the sample app uses queryFeatures or queryTask? Also, is the layer's visibility always turned off, so to be as close as possible to conditions in the App.

[MikeTschudi]

@CTLocalGovTeam The sample app tried both queryFeatures and QueryTask, and the layer's visibility was turned off before the queries.

[allisonmuise]

verified