[Bug]:

[SportsApps]

One or more sample matterport IDs / urls that reproduce the issue

https://my.matterport.com/show/?m=ksff5sJwfc1

Python version

3.12.4

Steps to reproduce

Try to see a downloaded space with advance mode

What went wrong

The space get freeze in the browser after launch the request for view the space downloaded

Workarounds

Hi. I have downloaded successfully a Matterport space with the new code, but if freeze and can see just only the first charging screen but never finish. Seem there are some error in the terminal:

macbookpro@MBP-de-David ~ % cd matterport-dl-main macbookpro@MBP-de-David matterport-dl-main % python3 matterport-dl.py ksff5sJwfc1 127.0.0.1 8080 View in browser: http://127.0.0.1:8080 127.0.0.1 - - [10/Aug/2024 01:39:27] "GET / HTTP/1.1" 200 - 127.0.0.1 - - [10/Aug/2024 01:39:27] "GET /?m=ksff5sJwfc1 HTTP/1.1" 200 - 127.0.0.1 - - [10/Aug/2024 01:39:27] "GET /JSNetProxy.js HTTP/1.1" 200 - 127.0.0.1 - - [10/Aug/2024 01:39:27] "GET /webgl-vendors/three/0.151.3/three.min.js HTTP/1.1" 200 - 127.0.0.1 - - [10/Aug/2024 01:39:27] "GET /js/browser-check.js HTTP/1.1" 200 - 127.0.0.1 - - [10/Aug/2024 01:39:27] "GET /js/showcase.js HTTP/1.1" 200 - 127.0.0.1 - - [10/Aug/2024 01:39:27] "POST /api/mp/models/graph?operation=GetModelPolicies HTTP/1.1" 200 - 127.0.0.1 - - [10/Aug/2024 01:39:27] "GET /locale/messages/strings_es.json HTTP/1.1" 200 - 127.0.0.1 - - [10/Aug/2024 01:39:27] "GET /js/254.js HTTP/1.1" 200 - 127.0.0.1 - - [10/Aug/2024 01:39:27] "GET /js/968.js HTTP/1.1" 200 - 127.0.0.1 - - [10/Aug/2024 01:39:27] "GET /js/627.js HTTP/1.1" 200 - 127.0.0.1 - - [10/Aug/2024 01:39:27] "GET /js/136.js HTTP/1.1" 200 - 127.0.0.1 - - [10/Aug/2024 01:39:27] "POST /api/mp/models/graph?operation=GetLayers HTTP/1.1" 200 - 127.0.0.1 - - [10/Aug/2024 01:39:27] "GET /locale/strings.json HTTP/1.1" 200 - 127.0.0.1 - - [10/Aug/2024 01:39:27] "POST /api/mp/models/graph?operation=GetModelViewPrefetch HTTP/1.1" 200 - 127.0.0.1 - - [10/Aug/2024 01:39:27] "POST /api/mp/models/graph?operation=GetLayers HTTP/1.1" 200 - 127.0.0.1 - - [10/Aug/2024 01:39:27] "POST /api/mp/models/graph?operation=GetLayers HTTP/1.1" 200 - 127.0.0.1 - - [10/Aug/2024 01:39:27] "GET /js/764.js HTTP/1.1" 200 - 127.0.0.1 - - [10/Aug/2024 01:39:27] "GET /js/648.js HTTP/1.1" 200 - 127.0.0.1 - - [10/Aug/2024 01:39:27] "GET /js/948.js HTTP/1.1" 200 - 127.0.0.1 - - [10/Aug/2024 01:39:27] "GET /js/217.js HTTP/1.1" 200 - 127.0.0.1 - - [10/Aug/2024 01:39:27] "GET /js/622.js HTTP/1.1" 200 - 127.0.0.1 - - [10/Aug/2024 01:39:27] "GET /js/969.js HTTP/1.1" 200 - 127.0.0.1 - - [10/Aug/2024 01:39:27] "POST /api/mp/models/graph?operation=GetSnapshots HTTP/1.1" 200 - 127.0.0.1 - - [10/Aug/2024 01:39:27] "GET /js/717.js HTTP/1.1" 200 - 127.0.0.1 - - [10/Aug/2024 01:39:27] "POST /api/v1/event HTTP/1.1" 200 - 127.0.0.1 - - [10/Aug/2024 01:39:27] "GET /js/625.js HTTP/1.1" 200 - 127.0.0.1 - - [10/Aug/2024 01:39:27] "GET /js/324.js HTTP/1.1" 200 - 127.0.0.1 - - [10/Aug/2024 01:39:27] "GET /js/477.js HTTP/1.1" 200 - 127.0.0.1 - - [10/Aug/2024 01:39:27] "GET /js/225.js HTTP/1.1" 200 - 127.0.0.1 - - [10/Aug/2024 01:39:27] "GET /js/519.js HTTP/1.1" 200 - 127.0.0.1 - - [10/Aug/2024 01:39:27] "GET /showcase-sdk/plugins/published/compass/1.0.14/plugin.json HTTP/1.1" 200 - 127.0.0.1 - - [10/Aug/2024 01:39:27] "GET /showcase-sdk/plugins/published/minimap/1.0.19/plugin.json HTTP/1.1" 200 - 127.0.0.1 - - [10/Aug/2024 01:39:27] "GET /showcase-sdk/plugins/published/salesperson-info/1.0.19/plugin.json HTTP/1.1" 200 - 127.0.0.1 - - [10/Aug/2024 01:39:27] "GET /showcase-sdk/plugins/published/quick-link/1.0.13/plugin.json HTTP/1.1" 200 - 127.0.0.1 - - [10/Aug/2024 01:39:27] "GET /js/529.js HTTP/1.1" 200 - 127.0.0.1 - - [10/Aug/2024 01:39:27] "GET /js/521.js HTTP/1.1" 200 - 127.0.0.1 - - [10/Aug/2024 01:39:27] "GET /webgl-vendors/three/0.151.3/libs/draco/gltf/draco_wasm_wrapper.js HTTP/1.1" 200 - 127.0.0.1 - - [10/Aug/2024 01:39:27] "GET /webgl-vendors/three/0.151.3/libs/basis/basis_transcoder.js HTTP/1.1" 200 - 127.0.0.1 - - [10/Aug/2024 01:39:27] code 404, message File not found 127.0.0.1 - - [10/Aug/2024 01:39:27] "GET /models/eb2134e0bb96420e83467deafe456159/assets/meshtiles//4.json?t=2-d3bb5d0eb497849aa5476176819c081611c612c7-1723304731-1&k=models%2Feb2134e0bb96420e83467deafe456159%2Fassets%2Fmesh_tiles HTTP/1.1" 404 -

mitchcaper, please. Could you have a look into it?

Maybe there isn't an issue and I'm doing something wrong...

[mitchcapper]

This model is a bit unique compared to some of the others, heavy 3d model data likely due to the (what I assume) is commercial construction data.

The problem was related do the 3d GLB file downloads. Not sure if it is a rate limit or just random failure but that code right now aborts on first error.

As a temporary work around I was able to download the model when I used a proxy for the traffic. Potentially due to the proxy slowing things down.

If you want to try this work around for now go to: https://www.telerik.com/download/fiddler and download "fiddler classic". Launch the program you do NOT need to enable "Capture Traffic" that tries to intercept all system traffic.

Do go into tools->options go to "https" tab and make sure it looks like:

Finally go to Actions button and click "Trust Root Certificate".

Finally go to the "Connections" it will say fiddler classic listens on port X.

Now runner the matterport download again but this time add --proxy 127.0.0.1:XYZ where XYZ is whatever port X was.

Note if you are ram constrained on the PC downloading in fiddler you will want to hit the black X in the toolbar periodically to clear out the connections its logged.

The model, given the 3d assets, is actually 17GB big.

Note: if it fails, you do not need to delete and restart the download you can re-run the same download command multiple times and hopefully it makes progress.

Each time it downloads it will give a progress report at the end like: Done, Total fetches: 17689 Skipped: 536 (3%) actual Request: 17153 (97%) Success: 16959 (96%) Failed403: 171 (1%) Failed404: 23 (0%) FailedUnknown: 0 (0%)!

The Success one hopefully goes up if not going up much it probably isn't making progress on each run.

I just added a new option so that after your first run you can disable downloading the primary assets, using --NO-MAIN_ASSET_DOWNLOAD it doesnt download things it has already downloaded but it does try resources that might exist.

You need to sync the repo for that change.

Leaving this open until I can figure out a proper fix.

[SportsApps]

Hi mitchcapper!! Thanks a lot for your answer and support!

I'm using mac so the Fidder Classic doesn't work on it. I have installed the Fiddler Everywhere one instead, configure it as the same and trust the root cert:

Then I have tried to download again using the: python3 matterport-dl.py "https://my.matterport.com/show/?m=ksff5sJwfc1" --advanced-download --proxy 127.0.0.1:8866

and re-run the existing matterport downloaded using: python3 matterport-dl.py ksff5sJwfc1 --proxy 127.0.0.1:8866

But in both cases seem to have a cert problem: File "/opt/homebrew/lib/python3.12/site-packages/curl_cffi/requests/session.py", line 1147, in request await task curl_cffi.curl.CurlError: Failed to perform, curl: (60) SSL certificate problem: unable to get local issuer certificate.

"CURLE_PEER_FAILED_VERIFICATION (60) The remote server's SSL certificate or SSH fingerprint was deemed not OK."

Anyway, if you are going to try to fire out another solution, I can wait until then. This is extremely important for my and appreciate a lot your help.

Best!

[mitchcapper]

Sorry yes you would need to install their SSL certificate temporarily. I went ahead though and added a --no-verify-ssl command line option so you don't need to do so. Just sync with the repo.

As you mention it is very important I re-ran the download myself just incase something happens to it before I get a proper fix in place and the proxy download doesn't work for you.

Hopefully in the next week or so I can get a better fix for this sort of digital twin to avoid the proxy requirements.

Technically any proxy may work (and fiddler anywhere may not) if it is a speed thing essentially a less efficient proxy may be key;)

[SportsApps]

Hi mitchcapper!! I have been able to redownload and open it. Unbelievable... I can see the matterport space now. I have no idea how you found the issue, but a million of thanks for your help! First, second and third rounds of beers are on my own if you visit Madrid.

I'll be pending for your solution in the repo. Thanks again!

[SportsApps]

I have tried to redownload the same space but another scan (I need to do the same every month). I takes ages to do it with the proxy, but after a wile I found that the proxy can be closed after the 3d model has been downloaded.

In other hand, I have tried to move the entire folder, including the downloaded spaces to a ssd due my mac is almost full. One the copy have been done, I have tried to launch the command for watch it on the nav, but it give me this error:

macbookpro@MBP-de-David matterport-dl-main % python3 matterport-dl.py 4qNWqMfoT3M 127.0.0.1 8080 Traceback (most recent call last): File "/Volumes/Crucial X8/matterport-dl-main/matterport-dl.py", line 1143, in openDirReadGraphReqs(os.path.join(BASE_MATTERPORTDL_DIR, "graph_posts"), pageId) File "/Volumes/Crucial X8/matterport-dl-main/matterport-dl.py", line 994, in openDirReadGraphReqs GRAPH_DATA_REQ[file.replace(".json", "")] = f.read().replace("[MATTERPORT_MODEL_ID]", pageId) ^^^^^^^^ File "", line 322, in decode UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb0 in position 37: invalid start byte

Can I execute it since and external SSD?

[mitchcapper]

it shouldn't matter what drive type it is on but it sounds like you sustained some corruption there. I would assume even if you moved it back to your other drive the same error would still happen. Try copying it from the one drive to the other a second time?

[SportsApps]

Ok, thanks. I'm trying copy the folder again. I'll let you know.

[SportsApps]

Same results here when I try to open the space from the SSD:

Traceback (most recent call last): File "/Volumes/Crucial X8/matterport-dl-main/matterport-dl.py", line 1137, in openDirReadGraphReqs(os.path.join(BASE_MATTERPORTDL_DIR, "graph_posts"), pageId) File "/Volumes/Crucial X8/matterport-dl-main/matterport-dl.py", line 993, in openDirReadGraphReqs GRAPH_DATA_REQ[file.replace(".json", "")] = f.read().replace("[MATTERPORT_MODEL_ID]", pageId) ^^^^^^^^ File "", line 322, in decode UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb0 in position 37: invalid start byte

I have unzip in a new SSD folder the matterport-dl-main. I have move the space into the downloads folder inside of this one. Tried to launch the view, but ge that error...

Maybe I need to download the space into the SSD?

[mitchcapper]

You can try changing the line in matterport-dl.py that currently reads: GRAPH_DATA_REQ[file.replace(".json", "")] = f.read().replace("[MATTERPORT_MODEL_ID]", pageId) to instead:

                print(f"Going to read graph file: {file} and replace pageid of: {pageId}")
                fn = file.replace(".json", "")
                text = f.read()
                text = text.replace("[MATTERPORT_MODEL_ID]", pageId)
                GRAPH_DATA_REQ[fn] = text

it will still throw an error but this time it should tell us right before it goes to do so what file it was trying to open (and also the exact line to confirm but i assume its the replace line).

If you open that file it mentions in the "graph_posts" directory in a text editor does it look fine?

I have no idea why your SSD would be having an issue like this especially if the same folder from your main drive with the same command works just fine.

Note python is whitespace critical meaning if when you replace the code it doesn't have things with equal spacing it will cry when you run it, essentially with the change the function should look like:

if you want to attempt that let me know what it says. The only other thing I can suggest is a different usb drive?

[SportsApps]

Hi mitchcapper. Thanks for your answer.

I have changed the line in matterport-dl.py:

Launch it again a show this:

macbookpro@MacBook-Pro-de-David matterport-dl-main % python3 matterport-dl.py 4qNWqMfoT3M 127.0.0.1 8080 Going to read graph file: GetModelDetails.json and replace pageid of: 4qNWqMfoT3M Going to read graph file: ._GetModelDetails.json and replace pageid of: 4qNWqMfoT3M Traceback (most recent call last): File "/Volumes/Crucial X8/matterport-dl-main/matterport-dl.py", line 1147, in openDirReadGraphReqs(os.path.join(BASE_MATTERPORTDL_DIR, "graph_posts"), pageId) File "/Volumes/Crucial X8/matterport-dl-main/matterport-dl.py", line 996, in openDirReadGraphReqs text = f.read() ^^^^^^^^ File "", line 322, in decode UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb0 in position 37: invalid start byte

The GetModelDetails.json looks exactly the same than the original one. Seem like this SSD create some ".-files". I saw it with some videos after reproduce them...

I have no another SSD but I'm going to try with a format SD and let you know.

Thanks again!!

[mitchcapper]

OK yes that solves it: Going to read graph file: "GetModelDetails.json" and replace pageid of: 4qNWqMfoT3M Going to read graph file: ._GetModelDetails.json and replace pageid of: 4qNWqMfoT3M

That second line there it is reading a file that starts with a period and then an underscore. I have no idea where that is coming from but it is not matterport-dl as far as am aware. Delete any ._* files and it should fit it. Matterport-dl expects only its own files in that directory and trys reading them all so some other app is adding some binary file there screwing it up.

[SportsApps]

I have done the change on the "GetModelDetails.json" but I still getting the same. I'm unable to find the ".-GetModelDetails.json", even showing all hidden files.

I have started create a new copy on an SD because the problem could be on the SSD. Here is too late now so I'm going to let the copy running and tomorrow I'll test it again. I let you know as soon as I can try.

Thanks a lot mitchcapper!!

[mitchcapper]

https://superuser.com/questions/212896/is-there-any-way-to-prevent-a-mac-from-creating-dot-underscore-files

[SportsApps]

Done!! You are a master! I have executed the dot_clean command on the graph_post and it works. Thanks sir! Really, I appreciate a lot your help.

[mitchcapper]

This download issue is fixed it should work without proxy now.

[SportsApps]

Great. It works!! Thanks a lot.