Public IBL new publication filter

iamamutt commented 2 years ago

https://github.com/vathes/ibl-navigator/blob/29cd8b9858b1ca86422978423c8369c2de2956da/backend/iblapi.py#L218

TODO:

[ ] Add new "Publication" drop down filter with the queries' "tag" attribute as the possible options. The tag will be used to restrict the table to only the sessions corresponding to the selected tag. @jverswijver @guzman-raphael
[ ] Copied over sessions to the public IBL database. Currently, tags can only be accessed from the SessionTag and Tag tables on the internal IBL database. @ttngu207 @iamamutt

Tag info

import datajoint as dj

dj.config["custom"]["database.mode"] = ""
dj.config["database.prefix"] = ""

from ibl_pipeline.acquisition import Session
from ibl_pipeline.behavior import SessionTag, Tag

tag_query = (Session * SessionTag.Tag * Tag) & "description = 'move-to-public'"

rs_data = (tag_query & "tag = 'Repeated-Site-2022'").fetch()
pd_rs_data = pd.DataFrame(rs_data)[["session_uuid"]]
print(pd_rs_data.to_markdown())

beh_data = (tag_query & "tag = 'Behavior-2022'").fetch()
pd_beh_data = pd.DataFrame(beh_data)[["session_uuid"]]
print(pd_beh_data.to_markdown())

`behavior.Tag()` table

tag	description
Behavior-2022	move-to-public
Behaviour Paper	alyx
Erdem's paper	alyx
Matt's paper	alyx
May 2021 pre-release	alyx
Repeated site	alyx
Repeated-Site-2022	move-to-public

`Repeated-Site-2022` tag

	session_uuid
0	41872d7f-75cb-4445-bb1a-132b354c44f0
1	88224abb-5746-431f-9c17-17d7ef806e6a
2	e535fb62-e245-4a48-b119-88ce62a6fe67
3	b03fbc44-3d8e-4a6c-8a50-5ea3498568e0
4	781b35fd-e1f0-4d14-b2bb-95b7263082bb
5	54238fd6-d2d0-4408-b1a9-d19d24fd29ce
6	a66f1593-dafd-4982-9b66-f9554b6c86b5
7	61e11a11-ab65-48fb-ae08-3cb80662e5d6
8	7691eeb3-715b-4571-8fda-6bb57aab8253
9	d0ea3148-948d-4817-94f8-dcaf2342bbbe
10	862ade13-53cd-4221-a3fa-dda8643641f2
11	72cb5550-43b4-4ef0-add5-e4adfdfb5e02
12	ff96bfe1-d925-4553-94b5-bf8297adf259
13	f312aaec-3b6f-44b3-86b4-3a0c119c0438
14	ecb5520d-1358-434c-95ec-93687ecd1396
15	15763234-d21e-491f-a01b-1238eb96d389
16	824cf03d-4012-4ab1-b499-c83a92c5589e
17	4a45c8ba-db6f-4f11-9403-56e06a33dfa4
18	03cf52f6-fba6-4743-a42e-dd1ac3072343
19	7af49c00-63dd-4fed-b2e0-1b3bd945b20b
20	c4432264-e1ae-446f-8a07-6280abade813
21	c51f34d8-42f6-4c9c-bb5b-669fd9c42cd9
22	56b57c38-2699-4091-90a8-aba35103155e
23	3638d102-e8b6-4230-8742-e548cd87a949
24	51e53aff-1d5d-4182-a684-aba783d50ae5
25	6f09ba7e-e3ce-44b0-932b-c003fb44fb89
26	746d1902-fa59-4cab-b0aa-013be36060d5
27	ebe090af-5922-4fcd-8fc6-17b8ba7bad6d
28	d9f0c293-df4c-410a-846d-842e47c6b502
29	754b74d5-7a06-4004-ae0c-72a10b6ed2e6
30	4b7fbad4-f6de-43b4-9b15-c7c7ef44db4b
31	dac3a4c1-b666-4de0-87e8-8c514483cacf
32	30c4e2ab-dffc-499d-aae4-e51d6b3218c2
33	b22f694e-4a34-4142-ab9d-2556c3487086
34	71e55bfe-5a3a-4cba-bdc7-f085140d798e
35	73918ae1-e4fd-4c18-b132-00cb555b1ad2
36	a8a8af78-16de-4841-ab07-fde4b5281a03
37	ee40aece-cffd-4edb-a4b6-155f158c666a
38	aad23144-0e52-4eac-80c5-c4ee2decb198
39	c7248e09-8c0d-40f2-9eb4-700a8973d8c8
40	7cb81727-2097-4b52-b480-c89867b5b34c
41	f140a2ec-fd49-4814-994a-fe3476f14e66
42	4b00df29-3769-43be-bb40-128b1cba6d35
43	dda5fc59-f09a-4256-9fb5-66c67667a466
44	d23a44ef-1402-4ed7-97f5-47e9a7a504d9
45	db4df448-e449-4a6f-a0e7-288711e7a75a
46	7f6b86f9-879a-4ea2-8531-294a221af5d0

`Behavior-2022` tag

	session_uuid
0	d7be2c7d-5dba-4083-9744-8c976e62fb43
1	fb0828de-914e-4259-a060-f58c3399d993
2	21628d97-cecc-48fc-bf08-f381d2efb26f
3	f415186d-58c2-421f-9915-c7c9f7ca6ef4
4	ac8214fd-e049-4692-a0f6-dd2be8858bc6
...
5809	84dff038-18a6-479b-926c-55ad95d53f7f
5810	d33baf74-263c-4b37-a0d0-b79dcb80a764
5811	259927fd-7563-4b03-bc5d-17b4d0fa7a55
5812	510b1a50-825d-44ce-86f6-9678f5396e02
5813	193fe7a8-4eb5-4f3e-815a-0c45864ddd77

ttngu207 commented 2 years ago

Thanks for putting this together @iamamutt

There are 5000+ sessions with Behavior-2022 tag to be published to the public ibl?

iamamutt commented 2 years ago

There are 5000+ sessions with Behavior-2022 tag to be published to the public ibl?

Yeah I noticed that too, over 5000 sessions which have datasets with the behavior tag at least. I'm trying to confirm with Mayo or Gaelle to see if this is right.

ttngu207 commented 2 years ago

As for the filtering, this can be fairly straight forward. Proposed changes to the code block around https://github.com/vathes/ibl-navigator/blob/29cd8b9858b1ca86422978423c8369c2de2956da/backend/iblapi.py#L275

sessiontags = acquisition.Session.aggr(behavior.SessionTag.Tag, session_tags='GROUP_CONCAT(tag SEPARATOR "; ")')

q = ((acquisition.Session() * sess_proj * psych_curve * ephys_data * subj *
              subject.SubjectLab() * subject.SubjectUser() * trainingStatus * sessiontags) & args & brain_restriction)

and the filtering can be something like ... & f'session_tags LIKE "%{selected_tag}%"'

iamamutt commented 2 years ago

As for the filtering, this can be fairly straight forward. Proposed changes to the code block around

https://github.com/vathes/ibl-navigator/blob/29cd8b9858b1ca86422978423c8369c2de2956da/backend/iblapi.py#L275
sessiontags = acquisition.Session.aggr(behavior.SessionTag.Tag, session_tags='GROUP_CONCAT(tag SEPARATOR "; ")')

q = ((acquisition.Session() * sess_proj * psych_curve * ephys_data * subj *
              subject.SubjectLab() * subject.SubjectUser() * trainingStatus * sessiontags) & args & brain_restriction)
and the filtering can be something like ... & f'session_tags LIKE "%{selected_tag}%"'

@jverswijver or @guzman-raphael Do you happen to know the database user that the public navigator is using?

We would like to be able to deploy a test version of the public navigator connecting to schemas with the test_ prefix, and implementing the new publication tag filters. Thinh has already copied over the data to those schemas.

datajoint-company / ibl-navigator