search

tlarsendataguy-yxdb / yxdb-net

A .NET library for reading the YXDB file format
MIT License
1 stars 0 forks source link

Crash when reading file with > 65535 rows #1

Open rmacfadyen opened 1 week ago

rmacfadyen commented 1 week ago

I have a yxdb file that I cannot share (proprietary) that crashes when reading on record 65535. A hugely suspicious number. It crashed trying to read a string value, with an overflow (iirc... sorry my bad).

The file is not corrupt. I generated it from an AMP/e2 file by saving it as a "regular" AlterYXDB file. Further I can use teh AlterYX Designer to export the file contents as CSV just fine.

The file contents are a handful of strings, a few doubles, and a spatial object.

I tried reading the file using the Python library and it too crashes (did not write out the record number).

Any thoughts on why it might be crashing at record 65535?

tlarsendataguy commented 1 week ago

It looks like the spatial index might be causing issues. I was able to create a crash using a file I generated. I will need to dig into the file format a bit to figure out how to handle the spatial index, correctly.

In the meantime, you should be able to generate a file that will work with the current version of this package by checking the 'No Spatial Index' checkbox on the output tool:

image

rmacfadyen commented 1 week ago

So glad you were able to repro!

For my needs, for this specific file, exporting to .csv was a good work around. Thanks though!

Do you think the issue could occur in any file with a spatial index? For a different task I've used the library to read files with spatial data and grabbed the geojson without issue. Only 700 records though.