Support non ascii characters in texture file names

[MrDChristop]

Hi I am having a gltf file with swedish UTF-8 texture file names like LÖVÅS and so on. The utility throws a UnicodeEncodeError: 'ascii' codec can't encode character u'\xd6' in position 19: ordinal not in range(128) We could either support it or detect these characters and replace them with latin ones. Here is the full error log.

Traceback (most recent call last):
  File "C:\Users\d_chr\Downloads\gltf2usd-master\Source\gltf2usd.py", line 726, in <module>
    convert_to_usd(args.gltf_file, args.usd_file, args.fps, args.scale, args.arkit, args.verbose, args.use_euler_rotation)
  File "C:\Users\d_chr\Downloads\gltf2usd-master\Source\gltf2usd.py", line 679, in convert_to_usd
    usd = GLTF2USD(gltf_file=gltf_file, usd_file=usd_file, fps=fps, scale=scale, verbose=verbose, use_euler_rotation=use_euler_rotation)
  File "C:\Users\d_chr\Downloads\gltf2usd-master\Source\gltf2usd.py", line 51, in __init__
    self.gltf_loader = GLTF2Loader(gltf_file)
  File "C:\Users\d_chr\Downloads\gltf2usd-master\Source\gltf2loader.py", line 115, in __init__
    self._initialize()
  File "C:\Users\d_chr\Downloads\gltf2usd-master\Source\gltf2loader.py", line 125, in _initialize
    self._initialize_nodes()
  File "C:\Users\d_chr\Downloads\gltf2usd-master\Source\gltf2loader.py", line 142, in _initialize_nodes
    node = Node(node_entry, i, self)
  File "C:\Users\d_chr\Downloads\gltf2usd-master\Source\gltf2\Node.py", line 8, in __init__
    self._name = '{0}_{1}'.format(self._name, node_index)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xd6' in position 19: ordinal not in range(128)

Thanks Dimi

[MrDChristop]

When renaming the textures and fixing the gltf file to have the new texture names that do not have ther UTF-8 characters the asset processes fine.

[kcoley]

@MrDChristop thanks for flagging this. I can see if we can check for Unicode characters and see if USD supports it. Otherwise we can go down the renaming route.

[kcoley]

@MrDChristop do you have a sample model?

[MrDChristop]

Utf8TestModel.zip Sure here you go.

[kcoley]

@MrDChristop thanks, I have a potential fix, but I’ll run it against your model when I have access to a machine.

[MrDChristop]

You can send me your fix and i can test it for you.

[kcoley]

@MrDChristop thanks! I made a PR fix for this. Feel free to test it out and let me know if this works: https://github.com/kcoley/gltf2usd/pull/77

[kcoley]

I went ahead and merged the change. Feel free to reopen if the issue comes up again

[MrDChristop]

Hi your solution does not work. I assume you did not run it through my model i send you. this is what it outputs

Found unresolvable external dependency 'baked_LYCKSELE_LÖVÅS_1_diffuse_crate_1.jpg'. (violates rule(s) 4)
Warning: in _AssetLocalizer at line 647 of c:\users\d_chr\downloads\usd-master\pxr\usd\lib\usdutils\dependencies.cpp -- Failed to resolve reference @baked_LYCKSELE_LÖVÅS_1_diffuse_crate_1.jpg@ with computed asset path @baked_LYCKSELE_LÖVÅS_1_diffuse_crate_1.jpg@ found in layer @c:/Users/d_chr/Documents/CrateVray/das.usdc@.
could not create das.usdz

[MrDChristop]

I also found out that if you want to use the usdz model for ARQuicklook in IOS for example on a website with the "ar" tag, so it can work on ios safari browsers in AR, you have to remove the non ascii characters. My webhost for example was not able to load a usdz model with these characters. LYCKSELE_LÖVÅS Which are the same characters used in the test model i send you. Please either consider replacing the non unicode characters for now or if you want i can retest any other solution.

[kcoley]

@MrDChristop I was able to convert your model on Mac OS, though I did modify the texture name in the glTF file since one character was different from the texture name on disk: sdf_vray_utf8.usdz.zip I changed baked_LYCKSELE_LÖVÅS_1_diffuse_crate_1.jpg to baked_LYCKSELE_LôVèS_1_diffuse_crate_1.jpg

But if it does not work on your OS or with the ar tag, I will do a text replacement to ascii instead.

[MrDChristop]

I use windows. The usdz file you send does not load the texture on windows. Running a "usdcat" command on the usdc file shows that it did not keep the utf8 non ascii characters.

You should not have to modify anything the texture file has the same name in the gltf file as on disk on windows.

Very strange. If you drag the gltf+bin+texture into this Web viewer it should work fine. The gltf file is ok. https://gltf-viewer.donmccurdy.com/

Replacing to ascii seems like the safest route

[kcoley]

@MrDChristop very interesting. So it seems the behavior with non ascii files are different on Mac OS and Windows.

I will try to do an ascii replacement, though the challenge is finding a way for Python to locate the texture file on disk based on the name in the glTF file that works on all OS. For instance, on Mac OS, it seems to map Å to è. I am not sure of the behavior on Linux as well. I'll dig into finding a solution for this.

[kcoley]

@MrDChristop I just merged another drop to master which now renames the unicode file to ascii. Feel free to give that a try. I had to implement this on Windows since Mac OS changes the filename of the texture on disk.