LorenFrankLab / spyglass

Neuroscience data analysis framework for reproducible research built by Loren Frank Lab at UCSF
https://lorenfranklab.github.io/spyglass/
MIT License
94 stars 43 forks source link

Export failure from retriction graph whith multiple restrictions to the same leaf #1170

Closed samuelbray32 closed 2 weeks ago

samuelbray32 commented 4 weeks ago

Describe the bug

To Reproduce In the franklab database:

from spyglass.common.common_usage import unique_dicts
paper_key = {"paper_id": "oct2024_testing_v4"}
table_restr = {"table_name":"`lfp_electrode`.`l_f_p_electrode_group__l_f_p_electrode`"}
table_key = {**paper_key, **table_restr}
ExportSelection() * ExportSelection.Table & table_key # contains 3 unique entries, with 3 values in `lfp_electrode`.`l_f_p_electrode_group__l_f_p_electrode`
ExportSelection().get_restr_graph(table_key).leaf_ft # has only a single entry in the table

Solution ideas

  1. In ExportSelection.get_restr_graph combine the uniques restrictions for the same table using a dj.OrList()
samuelbray32 commented 3 weeks ago

Additional note: the restriction strings placed in the ExportSelection.Table() entries do not all have balanced parentheses (saw multiple with dangling parentheses on the end. This causes issues whith the OR logic when applying them together as a list.

Code to sanitize these:

cleaned_restriction = (
        "("*restriction.count( "(" )
       + restriction
       + ")"*restriction.count( ")" )
)
CBroz1 commented 3 weeks ago

I ran update on the restrictions captured here to remove unbalanced parens, and found the leaf node updated appropriately. I've pushed an update to #1164 that should fix the unbalanced parens issue, and I've also rerun the Export.populate/resulting shell script. The _Populate.sql is updated in the relevant export folder