Azure / azure-remote-rendering

SDK and samples for Azure Remote Rendering
MIT License
106 stars 38 forks source link

[Conversion Problem] Conversion fails for asset with multiple input files #82

Closed robbbbbb closed 2 years ago

robbbbbb commented 2 years ago

Describe the problem

At some point in the last 2 months our method of converting certain models (which contain multiple files referenced by a single .gltf) has stopped working. We don't convert models like this that often, so we hadn't noticed the breakage and can't really narrow down exactly when it occurred.

Files that use a layout as simple as the following exhibit the problem:

<storagecontainer>/<dirname>/scene.gltf <storagecontainer>/<dirname>/scene.bin

And produce output .results.json files with contents like the following:

{"error":"2007","title":"The input asset is not valid.","details":{"import error":"GLTF: could not open referenced file \"scene.bin\""}}

The code on our side which uploads the files to an input storage container and constructs + issues the REST request to start the conversion hasn't changed during the time since this was working.

We've also recreated the issue using the Conversion.ps1 powershell script in this repo and via manually constructing and issuing REST requests. Our accounts use and always have used linked storage, but just in case we've tried both with and without providing the inputLocation.storageContainerReadListSas argument to the create request. We're seeing the same results in all cases.

The one method that doesn't seem to generate this issue, and which does work, is the latest version of the ARRT tool. Performing the equivalent operations to kick off a conversion with the tool produces a successful result.

Provide additional information

Below I'll provide a bunch of conversionIds, how they got generated, and the error/success results for each. Unless otherwise specified, they used this file layout, which is just this sketchfab file, downloaded using sketchfab's 'autoconvert GLTF' download option:

. ├── license.txt ├── scene.bin └── scene.gltf

  1. Direct REST submission using linked storage account d33167a4-f7a1-4b97-8d16-6ebcd278d855 Failed: {"error":"2007","title":"The input asset is not valid.","details":{"import error":"GLTF: could not open referenced file \"scene.bin\""}},
  2. Direct REST submission using SAS URI 602d96b6-7719-4257-bcd4-8e2020967711 Failed: {"error":"2007","title":"The input asset is not valid.","details":{"import error":"GLTF: could not open referenced file \"scene.bin\""}}
  3. ARRT submission 0b89cd8c-deeb-49b7-8649-129cd8643420 Succeeded!

Additional context

In addition to those above, here are the conversion IDs for an old conversion of similar assets (i.e. a multi-file model obtained from sketchfab with identical file layout). These were both submitted for conversion to the same ARR resource, with the same storage account, using the same code and the same original assets (a gltf zip of this sketchfab asset).

One succeeded, one failed. Unfortunately these conversion IDs, despite coming from the API originally and being stored in our database, no longer seem to be available for lookup in the conversion API (or something else has gone wrong on our end and those conversions were somehow created in a different region/account, but I feel like that's unlikely looking at our code + config changelog). Do finished conversions have some kind of maximum lifetime after completion that they're available for query?

  1. May 4th (success) 29aca865-5cd4-431f-87c5-564ebffa1bd7
  2. July 18th (failure) 5f00fc8b-1182-483f-8334-633102179cd0
jakrams commented 2 years ago

Hey @robbbbbb thank you for the very detailed description. This gives us a lot of information to try to reproduce and get to the bottom of the problem, and we'll come back to you as soon as possible.

Generally this kind of conversion is pretty standard for GLTF files and we are currently not aware of any issues. The only thing that comes to my mind right now, is that the ARRT tool generates an access token for the entire container, so that the conversion service has full access to everything. Code is here. Maybe your token is more restrictive? Of course our ps1 scripts should do the right thing regardless, so we'll investigate that use-case.

To answer your last question: Yes, the conversion IDs are only kept in the database for a limited time (either a week or a month, can't remember the details). So it's expected that you don't get data about them anymore.

Sorry for the inceonvenience!

robbbbbb commented 2 years ago

Thanks for the quick response @jakrams, our production code has always just used linked storage accounts rather than SAS URIs. The test I did using a SAS URI was was just a URI (container level, Read + List) I manually generated in the azure portal.

By the way, looking at the ARRT code you linked I noticed that it generates the token with Read + Write + List + Create privileges (see this line) rather than just the Read + List privileges specified here.

The Conversion.ps1 powershell script is similar: it generates a SAS token with full container privileges, including Delete (see this line)

Just to double check, I generated a SAS token with the same privileges used by the ARRT and submitted it via the REST API, but the conversion still failed, so I'm guessing Read + List are still valid and the issue is elsewhere.

jakrams commented 2 years ago

Just to let you know: we can reproduce the issue and are currently investigating what's going wrong. I'll keep you posted.

jakrams commented 2 years ago

It appears that a recent optimization broke multi-file conversions under certain conditions. We have a fix that we are currently testing and if all goes well, we will deploy it in the next few days. I'll keep this ticket open until then.

Thanks a lot for the very detailed report, it helped us find the issue relatively quickly.

robbbbbb commented 2 years ago

Awesome, thanks for the quick work @jakrams, I'll keep an eye on this issue 👍

jakrams commented 2 years ago

Hey @robbbbbb the fix for the conversion service has been deployed. Our tests have been successful, but it would be great, if you can give it a shot and let us know whether it works as expected for you now as well. You do not need to update your SDK version or anything, just run a new asset conversion.

robbbbbb commented 2 years ago

@jakrams I can confirm multiple file conversions are now working again for us, thanks for the fast resolution 🙏