microsoft / Application-Insights-Workbooks

Templates for Azure Monitor Workbooks
MIT License
562 stars 462 forks source link

Long file/path names being truncated in npm pack process? #1051

Closed gardnerjr closed 3 years ago

gardnerjr commented 3 years ago

at some point in the packaging/deployment process, it unpacks the templates to turn them into an npm package, and the long path/names and the build’s directory structure mean it exceeds some limits, either in tar/gz itself or in the build pipeline system:

D:\a\r1\a\npm-publish_azure_monitor_workbooktemplates-1.0.0-build.15583.tgz\azure_monitor_workbooktemplates-1.0.0-build.15583.tar\package\en-us\workbooks\Network Insights-ExpressRouteCircuitsWorkbooks-NetworkInsights-ExpressRouteCircuitMetrics-Minified

The “minified” file gets truncated and loses its suffix, that file was ...-Minified.json. (the path as above is 257 ch, which implies a 255 limit (not sure where the extra 2 are, but i want to say the actual path above doesn't actually have _ prefix/suffix but the build output might be adding that? or perhaps the drive specifier doesn't count as part of the length? unsure)

gardnerjr commented 3 years ago
  1. i updated the release pipeline to allow overwriting files when uncompressing the package from the build. it should never have to do that, but somehow the path limitations above ended up having multiple files end up being the same path - that at least unblocked releases

  2. i've updated the build output to not include the full package name+version, but to always have the build output a package named "build.tgz", which gets back around 100ch - but that didn't seem to fix it, so it isn't only limited on the release side

  3. the network insights team has changed the folder names for this template to bring it down and it isn't getting truncated in new builds

i'm still investigating if this is a limitation in npm pack in the folder it gets run? even stranger: there is another file with a LONGER path that does not get truncated?

gardnerjr commented 3 years ago

apparently this a limit of the 'tar' file format used inside the npm package's tgz:

Archive format, compatible with the V7 implementation of tar. This format imposes a number of limitations. The most important of them are: The maximum length of a file name is limited to 99 characters. The maximum length of a symbolic link is limited to 99 characters.

i'll have to add a check that verifies that limitation to the build process.

gardnerjr commented 3 years ago

this makes no sense. this one is truncated to 99 (wanted to be 104)

Network Insights-ExpressRouteCircuitsWorkbooks-NetworkInsights-ExpressRouteCircuitMetrics-Minified_

this one, also exceeds 99 (exactly 100):

Virtual Machines - Performance Analysis-Performance Analysis for a Group of VMs (Azure Preview).json

but is NOT truncated and works fine

gardnerjr commented 3 years ago

haven't seen this issue since we re-arranged things and added checks/errors, closing