Error when saving spike arrays from model that was ran twice

cmazzio commented 2 weeks ago

See model database and script for saving spike arrays below. When running saving_tau.py for CM43_Test2_qhcl (taste 2), I get the following error:

Screenshot from 2024-06-15 14-22-28

model_database.csv

cmazzio commented 2 weeks ago

Oh wait here's the saving_tau.py script saving_tau.txt

abuzarmahmood commented 2 weeks ago

So it seems like the script is running all the way to the end (line 65) and breaking at the saving part. My guess is that you might not have the directory setup where it is trying to save the numpy file.

Try adding the following right before where the saving begins (right before line 65) and make sure that the path it is trying to save the file exists. If the folder does not exist, you have to make it either manually or in the script before the file can be saved there print('/media/cmazzio/large_data/Change_point_models/'+exp_name[0]+'/'+exp_name[0]+'_'+model_id[0]+'_scaled_mode_tau.npy')

cmazzio commented 2 weeks ago

This is the error message I get when I add that print statement Screenshot from 2024-06-15 15-01-13

abuzarmahmood commented 2 weeks ago

Ok, that makes sense. Because there are multiple models which pass your conditions (table below), you have multiple paths so it's probably having trouble generating the save path correctly.

The rows below are what I'm left with after the above conditions. It might help to add the below lines to remove duplicates [LINK to lines]

fit_database.drop_duplicates()
fit_database.clear_mismatched_paths()

preprocess.time_lims | preprocess.bin_width | preprocess.data_transform | preprocess.preprocessor_name | model.states | model.fit | model.samples | model.model_kwargs | model.model_template_name | model.inference_func_name | data.data_dir | data.basename | data.animal_name | data.session_date | data.taste_num | data.laser_type | data.region_name | exp.exp_name | exp.model_id | exp.save_path | exp.fit_date | module.pymc3_version | module.theano_version -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- [2000, 4000] | 50 | None | preprocess_single_taste | 4 | 40000 | 20000 | {'None': None} | single_taste_poisson | advi_fit | /media/cmazzio/large_data/CM43/CM43_CTATest2_h2o_nacl_qhcl_240412_123531 | CM43_CTATest2_h2o_nacl_qhcl_240412_123531 | CM43 | 123531 | 2 | | gc1 | CM43_CTATest2_QHCl | 1003eda3 | /media/cmazzio/large_data/Change_point_models/CM43_CTATest2_QHCl/CM43_CTATest2_QHCl_1003eda3 | 6/2/2024 | 3.9.3 | 1.0.5 [2000, 4000] | 50 | None | preprocess_single_taste | 4 | 40000 | 20000 | {'None': None} | single_taste_poisson | advi_fit | /media/cmazzio/large_data/CM43/CM43_CTATest2_h2o_nacl_qhcl_240412_123531 | CM43_CTATest2_h2o_nacl_qhcl_240412_123531 | CM43 | 123531 | 2 | | gc1 | CM43_CTATest2_QHCl | 1c282bc0 | /media/cmazzio/large_data/Change_point_models/CM43_CTATest2_QHCl/CM43_CTATest2_QHCl_1c282bc0 | 6/2/2024 | 3.9.3 | 1.0.5 [2000, 4000] | 50 | None | preprocess_single_taste | 3 | 40000 | 20000 | {'None': None} | single_taste_poisson | advi_fit | /media/cmazzio/large_data/CM43/CM43_CTATest2_h2o_nacl_qhcl_240412_123531 | CM43_CTATest2_h2o_nacl_qhcl_240412_123531 | CM43 | 123531 | 2 | | gc1 | CM43_CTATest2_QHCl | 0a018752 | /media/cmazzio/large_data/Change_point_models/CM43_CTATest2_QHCl/CM43_CTATest2_QHCl_0a018752 | 6/2/2024 | 3.9.3 | 1.0.5 [2000, 4000] | 50 | None | preprocess_single_taste | 3 | 40000 | 20000 | {'None': None} | single_taste_poisson | advi_fit | /media/cmazzio/large_data/CM43/CM43_CTATest2_h2o_nacl_qhcl_240412_123531 | CM43_CTATest2_h2o_nacl_qhcl_240412_123531 | CM43 | 123531 | 2 | | gc1 | CM43_CTATest2_QHCl | 98dc7b91 | /media/cmazzio/large_data/Change_point_models/CM43_CTATest2_QHCl/CM43_CTATest2_QHCl_98dc7b91 | 6/2/2024 | 3.9.3 | 1.0.5

But because you have 2 models for 2 different numbers of states, you will have to add a constraint for what number of states you want, and then you have to add something to your code to pick only one of them (unless duplicates are otherwise deleted). I've added code to the file below that SHOULD do that.

saving_tau_updated.txt

Here is a visualization of the additions.

Let me know if you still get errors.

cmazzio commented 2 weeks ago

Thanks so much! I tried running the updated file (I just changed wanted states to 4, not 3) and I got this error: Screenshot from 2024-06-15 15-28-45

abuzarmahmood commented 2 weeks ago

Another edit. I didn't see the script already had something to deal with duplicates. We're already doing that in the previous edit when we select only the first row of the wanted frame. Try again and see what happens. Also added a print statement to facilitate potential future debugging.

saving_tau_updated_2.txt

cmazzio commented 2 weeks ago

I just got this error now:

Screenshot from 2024-06-15 15-42-41

abuzarmahmood commented 2 weeks ago

The original script was meant to extract the changepoint values and spike train en masse from a bunch of arrays (which I also failed to notice when we started editing this script). But now it will only work extract items from a single model at a time.

saving_tau_updated_3.txt

cmazzio commented 2 weeks ago

I got this error initially, but it seemed like it was just for the print statement so I commented it out and ran again and got the error in the next comment. Screenshot from 2024-06-15 16-07-20

cmazzio commented 2 weeks ago

latest error after commenting out print(num) statement Screenshot from 2024-06-15 16-08-42

cmazzio commented 2 weeks ago

This seems like maybe it's an issue again with my saving file lines?

abuzarmahmood commented 2 weeks ago

Yeah, I think if you get rid of the [0]s in the save_path, it should work (i.e. exp_name[0] --> exp_name, and model_id[0] --> model_id) because those variables are no longer lists, but are strings now

abuzarmahmood / pytau

Error when saving spike arrays from model that was ran twice #36