Implement verification and character saving functionality to address the Service Length Limitation issue, throw errors if [web address + folder + UUID/Smallkey + layer name] is too long #1
I am comfortable with item 4. I know we never implemented small keys in the project. We just used the first 8 characters of the UUID. FGP had concerns about this but the chances of this being the same are so slim. I would rather deal with that 1 in a million case then having 20 percent of services that don’t publish.
Would publishing the services under the first 8 characters of the UUID have any impact on RCS or RAMP integration? If not, please implement that approach.
If I understand correctly, that leaves us (88-14 = 64) characters for layer name. We could add some constraints at the time of shapefile upload to force shapefile names under 64 characters to ensure we don’t run into this again, and in the meantime communicate the limitation to our users.
How does that sound?
From: Bowerman, Dan (EC/EC)
Sent: December 23, 2015 2:35 PM
Subject: The PAGER service length issue
As you know, we’ve run into this service length issue a few times in the past couple of weeks. It appears there’s a limit to how long a service name can be in ArcGIS, which returns errors when shapefiles with long layer names are uploaded to PAGER for publication.
These seem to be related to Windows file path limitations, and since most ArcGIS server instances that PAGER is/will be deployed to are a black box, we should make some considerations on our end on how things get published.
There are 3 components to a service’s length:
The folder the service is published to (by default we have used “data_donnees” – 12 characters long)
The service name itself (i.e. 49deb8b2-10a6-4b4a-ad7c-9cbc2eda260b – always 36 characters long as this is based on the UUID of the metadata record)
The layer name underneath it, by default this is the name of the shapefile being published (i.e. “SIG_CAN_TrajectoiresVerifiees 1980-2009”)
There components combined make up the total length of the service. The sweet-spot, from my testing, appears to be 88 characters.
Under the current workflow, the overhead actual permits only a layer name of 50 characters. That is to say, we have 12 characters for the folder, a slash, then 36 characters for the UUID, then another slash (i.e “data_donnees/49deb8b2-10a6-4b4a-ad7c-9cbc2eda260b/”. The rest can be used for the layer name.
Where we start to run into problems on Production is when the layer name is too long. PAGER simply will not publish it, it cannot be published manually either. One example of a layer name that couldn’t be published is “releases-disposals-transfers-by-media-air-other-substances”. That’s 60 characters. 50 + 60 = 110, far above the 88-character limit. As I mentioned over OCS, I had to truncate the UUID to a smallkey value in order to make it publish correctly: “data_donnees/22abff18/releases-disposals-transfers-by-media-air-other-substances“ is 81 characters long, and therefore publishes correctly.
Suggestions and stuff already implemented:
Reduce “overhead” by using a “data” folder instead of “data_donnees”. This gives 8 more characters to work with. Already configured on Internal and External Production.
Idea: Go back to the smallkey approach, where we are using an 8 character identifier instead of 36 characters. This is what helped this morning. Frees up 28 characters.
Idea: Can the catalogue truncate a shapefile’s name upon upload to only be 58 characters long or less? Or reject shapefiles that are longer than this? May not be as desirable a solution.
Idea: A combination of 1 and 2 gives us a maximum of 86 characters to work with for a layer/shapefile name.
Lose the folder and keep services in the root ArcGIS services directory would free up an additional 5-13 characters.
Some food for thought. As PAGER keeps taking in large quantities of datasets, we are most definitely going to see this error come up again. I’m happy to discuss possibilities for changes on either sides of the workflow. I’m not able to find any official documentation about these limitations from Esri, this is anecdotal from our tests in Production over the past couple of weeks – but once we keep the folder+UUID+layer name within the 88-character limit, it works every single time.
Some correspondence with Catalogue team:
Hi Dan,
I am comfortable with item 4. I know we never implemented small keys in the project. We just used the first 8 characters of the UUID. FGP had concerns about this but the chances of this being the same are so slim. I would rather deal with that 1 in a million case then having 20 percent of services that don’t publish.
Would publishing the services under the first 8 characters of the UUID have any impact on RCS or RAMP integration? If not, please implement that approach.
If I understand correctly, that leaves us (88-14 = 64) characters for layer name. We could add some constraints at the time of shapefile upload to force shapefile names under 64 characters to ensure we don’t run into this again, and in the meantime communicate the limitation to our users.
How does that sound?
From: Bowerman, Dan (EC/EC) Sent: December 23, 2015 2:35 PM Subject: The PAGER service length issue
As you know, we’ve run into this service length issue a few times in the past couple of weeks. It appears there’s a limit to how long a service name can be in ArcGIS, which returns errors when shapefiles with long layer names are uploaded to PAGER for publication.
These seem to be related to Windows file path limitations, and since most ArcGIS server instances that PAGER is/will be deployed to are a black box, we should make some considerations on our end on how things get published.
There are 3 components to a service’s length:
There components combined make up the total length of the service. The sweet-spot, from my testing, appears to be 88 characters.
Under the current workflow, the overhead actual permits only a layer name of 50 characters. That is to say, we have 12 characters for the folder, a slash, then 36 characters for the UUID, then another slash (i.e “data_donnees/49deb8b2-10a6-4b4a-ad7c-9cbc2eda260b/”. The rest can be used for the layer name.
Where we start to run into problems on Production is when the layer name is too long. PAGER simply will not publish it, it cannot be published manually either. One example of a layer name that couldn’t be published is “releases-disposals-transfers-by-media-air-other-substances”. That’s 60 characters. 50 + 60 = 110, far above the 88-character limit. As I mentioned over OCS, I had to truncate the UUID to a smallkey value in order to make it publish correctly: “data_donnees/22abff18/releases-disposals-transfers-by-media-air-other-substances“ is 81 characters long, and therefore publishes correctly.
Suggestions and stuff already implemented:
Some food for thought. As PAGER keeps taking in large quantities of datasets, we are most definitely going to see this error come up again. I’m happy to discuss possibilities for changes on either sides of the workflow. I’m not able to find any official documentation about these limitations from Esri, this is anecdotal from our tests in Production over the past couple of weeks – but once we keep the folder+UUID+layer name within the 88-character limit, it works every single time.