Tiling issue with asset id: 21889 when uploading through API

[pablocbre]

Hi,

I'm facing a problem when uploading assets through the API. Let me lay out the situation:

• I have a CityGML with a building.

• Uploading that CityGML through the UI works fine. No problems, it gets tiled and shows on screen nicely.

• Uploading that same CityGML through the API works, but the tiling process fails and delivers a "We're having trouble with your data..." message.

• Fun fact: if I download the CityGML again from the troublesome asset Source Files' tab, and reupload it back again through the UI, the tiling process does work.

Hoping you can help me spot the error. BTW, the new API looks awesome. Keep up the good work.

[OmarShehata]

Thanks for reporting this issue @pablocbre. Just to confirm, can you share the route you're using/how you're making the request to the API?

[pablocbre]

Hi @OmarShehata. I followed the instructions found here. I used python for it.

I guess the relevant part here is the upload? This is the code I used for uploading. I'm using boto3.

client = boto3.client('s3',
                      aws_access_key_id=location['accessKey'],
                      aws_secret_access_key=location['secretAccessKey'],
                      aws_session_token=location['sessionToken'])

with open(file_to_upload, 'rb') as file:
    client.upload_fileobj(Fileobj=file,
                          Bucket=location['bucket'],
                          Key=location['prefix'] + name,
                          Config=S3Client.multipart_config)

Where S3Client.multipart_config = TransferConfig(multipart_threshold=0.1*GB).

What other specific details could be useful for you to know?

Thanks

[OmarShehata]

It looks like what's happening here is that it's correctly creating the asset but it's not actually uploading any data. The s3 bucket is actually empty.

I'm curious if it's an issue with reading the file there. Have you tried uploading a file by passing ina filename with upload_file?

[pablocbre]

Hi @OmarShehata. I don't think the problem is the file is not getting uploaded because I can actually see and download the files I used (see picture below):

And the fun thing is if I download that GML and reupload it through the UI, it works fine.

[OmarShehata]

@pablocbre yeah you're right, I see it now. I am able to tile this asset using the rest API example code in this repo here:

https://github.com/AnalyticalGraphicsInc/cesium-ion-community/tree/master/tutorials/rest-api

Can you show me how you're setting the sourceType option when making the request in Python? The sourceType for this should be CITYGML based on the doc here https://cesium.com/docs/rest-api/#operation/postAssets

[pablocbre]

Hi @OmarShehata, this is the body I posted:

           {'name': 'test2',
            'type': '3DTILES',
            'description': '',
            'options': {
                'sourceType': 'CITYGML'}
            }

[OmarShehata]

The server log suggests that it's not receiving a valid sourceType, but your POST body looks correct.

Can you post the full code that I can run (with your credentials removed) for making the request, uploading and launching the tiling job? A GitHub gist or just posting it here if it's not too long would work.

[pablocbre]

Hi @OmarShehata,

Posting my full code would be rather unconvenient since I'm trying to develop a quite generic client for your API.

I've synthesized the steps in this example. The problematic behaviour happens when using this code.

Note: Python3

import requests
import boto3
from boto3.s3.transfer import TransferConfig

metadata = {'name': 'test',
            'type': '3DTILES',
            'description': '',
            'options': {
                'sourceType': 'CITYGML'}
            }
MY_TOKEN = "TOKEN"
headers = {'Authorization': 'Bearer ' + MY_TOKEN}
filepath = "path/to/file"
name = "my_name"

response = requests.post('https://api.cesium.com/v1/assets', data=metadata, headers=headers)
response_body = response.json()

location = response_body['uploadLocation']

client = boto3.client('s3',
                      aws_access_key_id=location['accessKey'],
                      aws_secret_access_key=location['secretAccessKey'],
                      aws_session_token=location['sessionToken'])
with open(filepath, 'rb') as file:
    client.upload_fileobj(Fileobj=file,
                          Bucket=location['bucket'],
                          Key=location['prefix'] + name,
                          Config=TransferConfig(multipart_threshold=0.1 * 1024 ** 3))

response = requests.post('https://api.cesium.com/v1/assets/' + str(response_body['assetMetadata']['id']) + '/uploadComplete', headers=headers)

[OmarShehata]

Thanks for posting this @pablocbre ! This minimal code example helped a lot. Looks like there are two issues here. It works if you change:

response = requests.post('https://api.cesium.com/v1/assets', data=metadata, headers=headers)

to

response = requests.post('https://api.cesium.com/v1/assets', json=metadata, headers=headers)

I think this will make requests set the right content type to application/json. The other thing is that the uploaded file needs to have a .gml extension. So you need to also change:

name = "my_name"

to

name = "my_name.gml"

Let me know if that works for you.

[pablocbre]

Hi @OmarShehata, I'll try your suggestion as soon as possible. Just one quick question about the gml extension: does it mean ZIP files can't be uploaded through the API?

[OmarShehata]

ZIP files definitely can. ion will always unzip it first before running the tiling job. The extracted assets still need to have the right file extensions.

Generally anything you can do with the UI should work the same way with the API. I think the only reason you could download the source file from the failed tile job and re-upload it and have it works is because the original code wasn't setting the right content encoding for JSON when sending. I think if you upload a file without a .gml extension in the UI it'll give you the same warning/error.

[pablocbre]

Ok great, thanks a lot Omar. Will let you know if I manage to make it work correctly.

[pablocbre]

Up and running, thanks @OmarShehata !

[OmarShehata]

Thanks for confirming @pablocbre ! I made a note to clarify in the docs that the file extension does matter since this was not obvious to me as a user.

[pablocbre]

Great. And as always, thanks a lot for being so accessible and helpful!

[shanmugavel28]

Thanks for posting this @pablocbre ! This minimal code example helped a lot. Looks like there are two issues here. It works if you change:

response = requests.post('https://api.cesium.com/v1/assets', data=metadata, headers=headers)

to

response = requests.post('https://api.cesium.com/v1/assets', json=metadata, headers=headers)

Converting data to json in post request fixed option.sourcetype invalid issue.

Thanks @OmarShehata @pablocbre