search

petrbroz / svf-utils

Utilities for working with the SVF file format used by Autodesk Platform Services.
https://petrbroz.github.io/svf-utils/
MIT License
127 stars 55 forks source link

ECONNRESET error when transferring SVF files from Forge services #48

Closed wallabyway closed 2 years ago

wallabyway commented 2 years ago

Getting a random ECONNRESET errors when download SVF resources from BIM360, when using Azure step function...

Is this a concurrency issue, or rate-limit error, or apogee error ?

downloading-error

petrbroz commented 2 years ago

Just to make sure I understand this correctly: a customer is using forge-convert-utils in an Azure Function, trying to download SVF assets from Forge, and they're getting this ECONNRESET error when doing so?

If that's the case, I'd suggest getting more information about the specific Azure Function pricing plan, runtime version, and instrumenting the code with some more logs, incl. timestamps. And is the single function trying to download all the SVF assets, or just a subset of them? The reason I'm asking is, in certain cases the default timeout for Azure Functions can be as low as 5min: https://docs.microsoft.com/en-us/azure/azure-functions/functions-scale#timeout.

wallabyway commented 2 years ago

@petrbroz - I believe you noticed this problem with Azure functions and separated the SVF download part, from the convert part. And then used Azure Queue's to handle job tasks. That seemed to be the workaround. @petrbroz, perhaps you can share some of your code ?

petrbroz commented 2 years ago

I haven't shared the source code of that project intentionally because it includes some proprietary stuff and hacks (e.g., the gltf-to-usdz conversion is handled by making another HTTP request to a separate Heroku service running a custom Docker container...) so people wouldn't be able to reuse it anyways.

But I can at least share the architecture diagram, for inspiration:

azure-setup

petrbroz commented 2 years ago

It turns out that this issue was on the Azure Function side after all, so let me close this issue.

camerongibbs-eic commented 2 years ago

@petrbroz image function timeout is configured to 1 hour. The function is hitting the ECONNRESET well before timeout, during the period where forge-convert-utils is printing the 'Reading meshpack #.. Reading materials' messages.

We have a similar architecture where an https triggered function app puts a job to a queue. Then we have a queue triggered function app servicing the queue with a scale out of up to 10. It runs forge-convert-utils, giving the the urn and token so that it can do the download and conversion. The problem seems to occur during periods of high scale out when there are a lot of messages posted to the queue close together so the queue triggered app scales out.

petrbroz commented 2 years ago

Thanks Cameron. How about separating the SVF download from the actual conversion? If you download the SVF using a separate Azure Function (and perhaps even using a custom code that could handle the ECONNRESET errors better), you could then use forge-convert-utils to translate a local folder with the SVF data instead.

camerongibbs-eic commented 2 years ago

@petrbroz thanks, we'll look into separating the download and conversion into separate stages when we get through the items on our roadmap. If you want to share the download code that would be appreciated, though I see it's a fairly easy to do step using the modelDerivativeClient.

petrbroz commented 2 years ago

The Azure function that's responsible for downloading the SVF is basically using the SvfDownloader class from forge-convert-utils, as demonstrated here: https://github.com/petrbroz/forge-convert-utils/blob/develop/samples/download-svf.js.