AllenInstitute / datacube

Other
0 stars 1 forks source link

projection data returned "NaN" from latest deployment including latest data refresh #96

Closed shus2018 closed 5 years ago

shus2018 commented 6 years ago

tdatacube was enabled auto-deployment on each Thursday night. There are some regressions from yesterday's deployment, it was OK yesterday, possible due to data refreshed and there are bad data retrieved and caused the issue.

e.g. tdatacube returned projection data "Nan": http://tdatacube:8080/mouseconn/data/projection_map/target?seedPoint=6600,5400,4800&startRow=25&numRows=50

e.g. devdatacube returned nothing: http://tdatacube:8080/mouseconn/data/projection_map/target?seedPoint=6600,5400,4800&startRow=25&numRows=50

e.g. prod datacube OK:

http://datacube1:8080/mouseconn/data/projection_map/target?seedPoint=6600,5400,4800&startRow=25&numRows=50

shus2018 commented 6 years ago

interesting, I re-generated conn data to a different folder and restart/reload data, it is working now. I will keep both data to debug next week in case. Possible storage mount related? We need to make sure.

http://tdatacube:8080/mouseconn/data/projection_map/target?seedPoint=6600,5400,4800&startRow=25&numRows=50

chrisbarber commented 6 years ago

The connectivity generation script in particular might not behave as expected if data files are already present. For now the scripts should be run with non-existent or empty data directories, and in the future an error or warning could be issued if that is not the case.

shus2018 commented 6 years ago

seeing this issue one more time, restart auto-fixed the issue. :-)