search

neurodata / ndstore

code for storing neurodata images and image annotations
http://neurodata.io
Apache License 2.0
38 stars 12 forks source link

cubes that are all 0 appear to be stored in the db #372

Open alexbaden opened 8 years ago

alexbaden commented 8 years ago

I was running the npz -> blosc conversion script and kept getting failures when trying to loop over the entire dataset.

After a bit of debugging, it appears that there are some cubes in the database that are all zeros...

I added a print statement for any cuboid that has isNotZeros() evaluate to false. It prints the x,y,z cube coordinates and the zindex.

Zero cube: 6, 0, 0 - 72

I then checked the database for this cube, and found it existed in the DB (even though the statement isNotZeros() returned false). 

mysql> select zindex from NR2B_9thA_res2 where zindex = 72;
+--------+
| zindex |
+--------+
|     72 |
+--------+
1 row in set (0.00 sec)

And, sure enough, if we multiply out the cuboid coordinates by the cube dimension (e.g. 384,0,0 and check the cutout, I get an error...

nd/sd/kristina15/NR2B_9thA/xy/2/384,440/0,100/1/

But, a cutout inside a cuboid that didn't return zeros works fine...

nd/sd/kristina15/NR2B_9thA/xy/2/100,200/0,100/1/

(note: this project on synaptomes has been fully converted already regardless of cube state, see below)

Here's a SQL dump of this table: https://www.dropbox.com/s/s5b8ookx6bq8sk2/NR2B_9thA_res2.sql?dl=0

The dataset is kristina15, channel is 16-bits.

I don't want to delete this zero cuboids until we've verified that there actually is no data in them. In the meantime, I changed the convert code to also convert 0 cuboids. That will at least let us look at the data and move forward with TRA related goals, and seems to be working. Was curious if there were any thoughts as to what is going on...

@randalburns @perlman

alexbaden commented 8 years ago

I can already se a problem with the above approach, I think. Converting the "0" cuboids is probably creating a lot of 0 cuboids...

Might be worthwhile to write a script that does a single pass search, finds any stored 0 cuboids, and deletes them from the DB. Happy to investigate if everyone else agrees.