Add the "Extract" capability to layer metadata

[jkerr5]

Can we add "Extract" to the capabilities property at the service and layer level metadata (the templates)? I don't see what the export capability requires beyond what the query capability provides but it appears to be used as a marker for tools like ArcGIS Pro to know if the user can export the data from the service.

If you don't want to add the capability for all services, can you support a flag that we can set in the configuration somehow?

[dmfenton]

This would have to be a flag. Exporting data into downloadable file formats is a heavy-duty feature

[jkerr5]

Do you have documentation that defines the behavior of the "Extract" capability? From what I can tell, it IS just a flag that tells feature service clients that they can download everything. That's what ArcGIS Pro does.

In other words, the "Exporting data into downloadable file formats" feature is implemented on the client side, not the feature service. Unless I'm missing something (which is quite possible!).

[dmfenton]

I need to go back and check whether export/extract are two different things.

[jkerr5]

Sorry, this is supposed to be the "Extract" capability. I'll correct the title and description. ArcGIS Pro has an "Export Features" menu that is disabled unless "Extract" is in the capabilities of the layer.

[dmfenton]

If that's the case then it is definitely heavy duty.

https://doc.arcgis.com/en/arcgis-online/manage-data/use-hosted-layers.htm#GUID-47A1D795-B330-45D7-89F7-9203A99E6924

Export data from hosted feature layers
You can export CSV files, shapefiles, a file geodatabase, GeoJSON files, feature collections, KML, or a Microsoft Excel file from a hosted feature layer if one of the following is true:

You own the features.
You are an administrator for your ArcGIS Online organization.
You aren't the hosted feature layer owner or the administrator, but the owner or administrator has configured the hosted feature layer to allow others to export the data.
This setting can be changed on the item page Settings tab by checking the Allow others to export to different formats check box under Export Data.

Only nonspatial attributes are exported to a CSV or Excel file when you export a line or polygon layer from a hosted feature layer. When you export a point layer to a CSV or Excel file, latitude and longitude values for the points are exported.

If the layers in the hosted feature layer contain metadata, the metadata is included if you export to a shapefile or file geodatabase.

When you export from a hosted feature layer view, only the data included in the view definition is included in the exported file.

All exported items are stored in the root folder of My Content.

Follow these steps to export data from the details page of a hosted feature layer or hosted feature layer view:

Sign in and open the item page for the features you want to export.
If you own the feature layer, click Content > My Content and click the item title.
If you do not own the feature layer, search for the layer, and click the feature layer name in the search results list.
If you want to export individual layers, go to the Layers section of the Overview tab, click Export To under the layer you want to export, and choose the format you want to export. If you want to export all the layers in the hosted feature layer, click the Export Data button on the Overview tab and choose the format you want to export.
Export to Shapefile—Creates a compressed file (.zip) containing a shapefile for each layer and its associated metadata (if present) that you export. You can download the file and save it to your computer.
Export to CSV file—Creates a comma-separated values file when you export from a layer. You can open the file or save it to your computer. If you export all layers to a CSV file, a CSV collection is created, which is a .zip file containing one CSV file per layer. You can download the .zip file and save it to your computer.
Export to KML—Creates a KML service you can open in Map Viewer. When you export all layers to KML, a KML collection is created. This collection is a .zip file containing one KML file per layer.
Export to Excel—Creates a Microsoft Excel spreadsheet. You can open the file or save it to your computer. If you export all layers to Excel, each layer will be a separate sheet in the spreadsheet.
Export to FGDB—Creates a .zip file containing a file geodatabase. The file geodatabase contains a feature class and its associated metadata (if present). You can download the .zip file and save it to your computer. Note that the .zip file uses the name you specify for the Title, but the geodatabase name is randomly generated, and the feature class has the same name as the layer you exported.
Export to GeoJSON—Creates a GeoJSON file containing definitions for all layers you export. You can download the file and save it to your computer.
Export to Feature Collection—Creates a feature collection item you can open in Map Viewer.
Choose Generalize features for web display to optimize the layer for web apps. You can only generalize features from layers published in the WGS 1984 Web Mercator (Auxiliary Sphere) coordinate system. Note that exported feature collections that are generalized for web display do not work in desktop and mobile apps.

Alternatively choose Keep original features if you need to maintain all the precision in your data, or if you intend to use the feature collection in desktop or mobile apps.

[jkerr5]

Still, I believe that the saving to different file formats is up to the consumer of the service, not the service itself. I have yet to find a specification that describes behavior for "Extract" that is beyond what "Query" provides.

[dmfenton]

It's not well documented, but that's not the case. The service itself generates the file using the sync API: https://developers.arcgis.com/rest/services-reference/sync-overview.htm

[jkerr5]

Pro doesn't use that as far as I can tell (at least not for what I am looking at). If "Extract" is in the capabilities, it will pull all the data from the service using the query endpoint and save to the formats it supports.

[jkerr5]

Also the "Sync" capability is called out separately from "Extract": https://developers.arcgis.com/rest/services-reference/feature-service.htm Are you sure these are connected capabilities?

[dmfenton]

It may mean different things to different consumers which is a bit of a problem. To ArcGIS Online it means that data can be extracted via the sync API

[jkerr5]

Yeah, that is problematic. What does the "Sync" capability mean then?

[dmfenton]

To my knowledge, there isn't one.

[jkerr5]

According to https://developers.arcgis.com/rest/services-reference/feature-service.htm

The capabilities property returns Create, Delete, Extract, Query, Update, Sync, and Uploads capabilities.

And then there's another flag for sync

syncEnabled will be true to indicate the support for sync.

So, back to the original request then: Can you add the ability for the "Extract" capability to be added to the "capabilities" properties so that the Export Features feature of Pro can be used against a Koop feature service?

[dmfenton]

No question we can add something to metadata to allow you to set this.

[jkerr5]

Thanks @dmfenton I found some docs on how Extract and Sync are connected here: http://enterprise.arcgis.com/en/server/latest/publish-services/windows/editor-permissions-for-feature-services.htm

Extract allows editors to extract copies of the data to a file geodatabase or SQLite database using custom applications that use the createReplica REST operation with the syncModel parameter set to none. This operation is disabled by default.

And following the docs for createReplica, it says this

syncModel=none can be used to export the data without creating a replica. Query the supportsSyncModelNone property of the feature service to see if this model type is supported.

So, Pro must be doing something different if a service has the Extract capability but is not enabled for sync.

Thanks for your patience working through this.

[rgwozdz]

@jkerr5 will the addition made in https://github.com/koopjs/FeatureServer/pull/120 cover your needs here?

[jkerr5]

@rgwozdz I believe that should work.

[rgwozdz]

@jkerr5 FeatureServer 2.16.0 was just published and supports addition of Extract to the layer info capabilities. I added some documentation as well. You can affect the change via a provider by adding capabilities = { extract: true } to the GeoJSON that gets passed into the getData callback.