search

melowntech / workshop

Workshop of Melown 3D stack
10 stars 3 forks source link

Core dump - 'vtslibs::storage::BadFileFormat' #35

Open Tekn1ck opened 2 years ago

Tekn1ck commented 2 years ago

Hi @vaclavblazek

I am having a big issue with the VEF2VTS tool. The VEF2vts tool loads all the windows and splits the tiles fine (it seems) but then about 1-2% of the way through generating the tiles it just throws the error below and crashes.

2021-11-09 22:48:13 I3 [850(tile:16-58907-56488)]: Generated tile #1905 of 254506 ( 0.75 % done): 16-58907-56488 (epsg27700, extents: 541321.192481,179184.067720:541331.920247,179202.757555) [mesh,atlas]. {encoder.cpp:process():304} terminate called after throwing an instance of 'vtslibs::storage::BadFileFormat' what(): File "/mnt/VTS_Processing/vef2vts_output/Project/tmp/0/c5/16-0001842-0001766.tiles:18,0,0" is not a VTS simplemesh file. @{tmptileset.cpp:loadSimpleMesh():180} Aborted (core dumped)

I have done all the usual checks - manifest has the correct number of tiles versus LODS and the number of tile folders is correct and everything matches. I am now going through and trying to do the tiles in smaller batches.

Can you give me any pointers as to what may be the reason for this and what I can do to troubleshoot the issue and isolate the cause?

Thanks

vaclavblazek commented 2 years ago

Pls, can you send me the file mention in the exception, i.e.: /mnt/VTS_Processing/vef2vts_output/Project/tmp/0/c5/16-0001842-0001766.tiles in this case.

Tekn1ck commented 2 years ago

It is a bit large, like 5gb and we are under NDA so it is hard to send this to you. I just did a smaller number of tiles and it worked - so I guess it could be something in the model files themselves. Could it be tiles with not enough geometry at the lowest LOD level for the splitting to have worked properly?

vaclavblazek commented 2 years ago

No, this is something wrong in the temporary data that are created during conversion. Which is very, very strange.

Tekn1ck commented 2 years ago

hmm - really hard - I will see if I can replicate the issue on a smaller dataset and see if I can get you that to work with - how would I get it to you?

vaclavblazek commented 2 years ago

Well, first, I guess the tiles file mentioned in the error is 5 GB, it should be way, way less. You may send it via drobpox or some other service. What distro and vef2vts tool version do you use?

Tekn1ck commented 2 years ago

Can I send a private message on here?

Ubuntu 18.04 LTS Latest version of vef2vts I believe I did an apt update before I started processing.

vaclavblazek commented 2 years ago

I'm not sure there is a PM support on github. You may send me an email (my name.surname@melowntech.com).

We have not made recent packages for bionic (18.04), due to some problems we switched to focal (20.04). How (in)convenient is to upgrade for you?

Tekn1ck commented 2 years ago

ok great - so it is not too hard - we are just running it on a VM. If we did an upgrade to 20.04 would that work - or better to do a fresh install?

Will pop you an email.

vaclavblazek commented 2 years ago

IMHO fresh install would be better -- no need to mess with different repos and their incompatibilities.

Tekn1ck commented 2 years ago

OK thanks,

Well I will wait first to see if you find anything before we do a bit overhaul.

Nick

[note: deleted personal info]

vaclavblazek commented 2 years ago

Aha, now I know :man_facepalming:. The underlying file format uses 32-bit indices and your's has 4.3 GB. Which cannot fit in 32-bit integer.

Now, the question is: why is this file so enormous? It contains up to 1024 tiles (cannot tell details now since the pointer to content index is nonsense).

How big are other .tiles files?

NB: upgrade is not gonna help here, but I'd recommend it anyway.

Tekn1ck commented 2 years ago

They vary from a fes mb through to gigabytes of data. It is late here today. But I can send you a directory tree tomorrow so you can see.

On Wed, 10 Nov 2021, 23:51 Václav Blažek, @.***> wrote:

Aha, now I know 🤦‍♂️. The underlying file format uses 32-bit indices and your's has 4.3 GB. Which cannot fit in 32-bit integer.

Now, the question is: why is this file so enormous? It contains up to 1024 tiles (cannot tell details now since the pointer to content index is nonsense).

How big are other .tiles files?

NB: upgrade is not gonna help here, but I'd recommend it anyway.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/melowntech/workshop/issues/35#issuecomment-965102396, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQGXPSK5D2LMG5YJ6TUORZLULJTF3ANCNFSM5HW6DFSQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

vaclavblazek commented 2 years ago

Hi, can you pls install package vts-tools version 1.99 from our repository (focal only). There's a brand new option --tweak.binaryOrder that sets size of single archive file in tiles (square, with edge 2^binaryOrder, i.e. total area in tiles is 4^binaryOrder). Try setting it to 4. I hope it'd help, at least to process data.

Tekn1ck commented 2 years ago

Thank you Vaclav,

I will let you know how I get on

Nick

On Mon, 22 Nov 2021 at 23:04, Václav Blažek @.***> wrote:

Hi, can you pls install package vts-tools version 1.99 from our repository (focal only). There's a brand new option --tweak.binaryOrder that sets size of single archive file in tiles (square, with edge 2^binaryOrder, i.e. total area in tiles is 4^binaryOrder). Try setting it to 4. I hope it'd help, at least to process data.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/melowntech/workshop/issues/35#issuecomment-975451641, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQGXPSJVEOZVB24AM4CBQJLUNIWTBANCNFSM5HW6DFSQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

--

Nick Pickop Group Director

Unit A, 2/F, Gold King Industrial Building, 35-41 Tai Lin Pai Road, Kwai Chung, New Territories, Hong Kong

Mob Primary(AUS) +61 412 731 029

Mob(Hong Kong): +852 9370 7836 Mob(UK): +44 7790 904579 Mob(Singapore): +65 9650 3805

LONDON • LISBON • HONG KONG • SINGAPORE

www.tekuchi.com