AllenInstitute / visual_behavior_analysis

Python package for analyzing behavioral data for Brain Observatory: Visual Behavior
Other
21 stars 6 forks source link

return failed experiments when not filtering for release_data_only #796

Closed matchings closed 2 years ago

matchings commented 2 years ago

This PR adds passed_only=False to the SDK call to get the experiments table in get_filtered_ophys_experiment_table() to make sure that failed experiments are also returned when release_data_only=False and include_failed_data=True.

This passed_only flag was recently added to the SDK with the default being True, which has caused much confusion (see https://github.com/AllenInstitute/AllenSDK/issues/2243). I have requested that instead they make a published_only flag and set it to False by default so that users can do their own filtering, which is typically what you want to do when loading from lims. Otherwise we can just use the S3 cache to get published data.

The PR also sets add_extra_columns to False by default because it is slow and most of the extra columns that we had been adding in the past are now part of the SDK experiments table itself. The remaining extra columns are primarily for detailed QC evaluations and dont need to be there by default.

matchings commented 2 years ago

Can you make passed_only be an input argument for get_filtered_ophys_experiment_table()? I'm ok with it defaulting to True, it just would be nice to have the ability to control the behavior

That would be the equivalent of setting the existing argument include_failed_data to False. Do you want me to change the name of the argument?

Also, I'll point out that setting include_failed_data to False gives back all passed experiments in lims. To get only published experiments you would also set release_data_onlyto True.

alexpiet commented 2 years ago

Oh duh. Nevermind