Extending Alyx

[luiztauffer]

Alyx is super handy and it’s great that so many models are already in place! I’m working to extend it to cover for needs of experiments with non-human primates, which will be basically creating some new tables and possibly extending a few existing ones.

My first approach is to create a new app (a new directory inside /alyx) similar to actions, misc, subjects… in a way to make it modular and as separated as possible from the original code. However, to really integrate the changes requires manually modifying some central project files, such as urls.py, settings.py and base.py.

My question is: is there any plan or interest in making a plug-and-play scheme for new models to be integrated, as per lab-needs? Am I being naive thinking that this wouldn’t be so difficult to do?

[rossant]

I think it would be a good idea to make alyx more modular and allow labs to develop their own extensions easily. It should be possible to do that without changing to much of the core files. I imagine one could have multiple urls.py files and import them from a single "master" file.

Could you be more specific on what exact features and schemes you would like to add in your version of alyx?

[luiztauffer]

Hi @rossant

It should be possible to do that without changing to much of the core files. I imagine one could have multiple urls.py files and import them from a single "master" file.

Exactly!

Could you be more specific on what exact features and schemes you would like to add in your version of alyx?

we are in the first steps of designing the models, so a lot might change, but here are some ideas we are likely to implement:

• BehavioralTasks table, to serve as a ForeignKey to Session. This should be useful to differentiate and filter by experimental tasks executed by the animals

• FoodConsumption table to control subjects food consumption over time, similar to Weighing table

• ConfigFileRecord (similar to FileRecord table) to map recording system’s configuration files for each Session

• ImagingFileRecord to map files from imaging recordings (MR, Xray, Ctscan), these are not bind to Sessions, but to Subjects

• Probably an extension of Subject (NHPSubject or something…) to account for NHP specifics

• Probably an extension of Sessions to accommodate some of these extensions

[oliche]

Hi Luiz a few comments:

• we've already extended the ephys sessions from sessions, in this case the underlying models are the same and only the admin interface is different, if you want to have a look in actions/admin.py.

• for imaging/histology our strategy is to create one histology session per subject to which we link concerned datasets. The procedures FK field of the session allows querying such sessions. We thought it would be simpler to always link data files to session objects.

• the config file record file could be registered as a dataset type on your database (we have such config files as well but they are really rig dependent)

• for the task, we explicitly chose to have a free text field task_protocol in the session table. We concatenate the task name and the version number in there. This is free text but only filled by a machine (it's read-only in the admin interface). We run our queries on this so far.

I think we could accomodate the FoodConsumption table. Note that there is already a Food table in the misc app to which we should link.

[bendichter]

Hey all, I'm also part of this effort. Thanks for getting the ball rolling, @luiztauffer and thanks for engaging so promptly, @oliche and @rossant! We are thinking about how we might be able to use Alyx for the Buffalo Lab, and thinking about how we might be able to use it more generally, extending it for the specific needs of different labs while keeping a common core. This is all really valuable information. I think there are two inter-related issues here:

1) What features should we add directly into this repo, (e.g. perhaps FoodConsumption), and what features should we separate out into a module that can add features on top of alyx for specific use-cases? 2) What is the best way to implement these modules? Are there small infrastructural adjustments to this repo that would make it easier to construct modules? Would there be a different strategy for adding tables vs. adding fields to an existing table?

for imaging/histology our strategy is to create one histology session per subject to which we link concerned datasets. The procedures FK field of the session allows querying such sessions. We thought it would be simpler to always link data files to session objects.

So would you recommend having all of the imaging be stored in Sessions objects that have procedures that indicate MR, Xray, and CT scan? Then we could link to the data file within the Session item. That could work.

I'm curious why you chose to make task name and version free text. I understand that if it is only ever written by a machine you probably won't have issues with standardized names, but still I think this could be limiting. In a standard lab setting, where you have many different tasks being run all the time and being constantly developed, this is probably going to be the # 1 thing that needs to be recorded well and that users are going to want to query. If there is any piece of data that might devolve into chaos if some structure is not defined, I think it would be task. (though I understand that this may not be the case for IBL, where tasks are much more standardized). We have been thinking about building additional infrastructure around tasks:

• Tasks would have required description and version
• Each task version would have a required change log
• There would be task categories, e.g. "y-maze," "free exploration," etc. that we facilitate querying
• Task features, e.g. is_calibration, is_auditory, etc. (possibly less or more as we iterate with the lab)

I'm not as experienced in schema design, but I think some of these might be difficult to do without a full-fledged Task table. I understand that these types of records may be out of scope for core alyx, so what would the best strategy be to add them as modules?

[oliche]

For extending procedures we would add records to the Procedures tables (MR, Xray, CT...). Those are general enough that I would even argue they could be added as fixtures (init models in json).

Those 2 changes, ie. FoodConsumption and a Task table are clearly within the scope for core Alyx.

For the task we didn't rule out the idea of a new table, we just started small to get some mileage. The IBL tasks are quite well defined with their change logs, so moving to a table architecture is always possible.

So far we managed to not fork the core Alyx, but I have to admit it is not entirely clear to me how to implement plugins or modules. Doing so most likely rules out extending the models of existing apps for non-core functionalities. But the new app approach would allow quite some freedom. Each plugin app would confine its own urls (as the experiments app does) and rebuild the url file. Also need to find a strategy to avoid merge hells for the settings files in the repository.

We have a weekly review on Thursday, I suggest to discuss those items there and update afterwards ?

[bendichter]

@oliche

Those 2 changes, ie. FoodConsumption and a Task table are clearly within the scope for core Alyx.

Great! We would be happy to work with your team to make additional tables in a way that would be generally useful. Ideally we could be able to agree on a general structure that could meet the needs of several different labs. If not, we can always revisit the idea of building small specialized extensions/modules for specific use-cases. Through our work with NWB, we've found that this is generally the best approach- try to build general solutions as much as you can, and also support an extension framework for when the needs fall outside the scope of the general community.

We have a weekly review on Thursday, I suggest to discuss those items there and update afterwards ?

Also great! I also want to make sure @nbonacchi sees this, as we've been mostly in touch with him about this project and he may have some additional context to add to your convo on Thursday.

[oliche]

@nbonacchi did you see this ;-)

[oliche]

Hello following up on this, let's make those 2 tables to start with ?

If you envisioned contributing as developer, I'd recommend starting with the models on a new branch as it's a good first feature, and once we agree on the tables I propose to take care of the admin interface and rest endpoints/filters unless you are already familiar with Django. @nbonacchi you had some suggestions on the Task table design ?

Usually when working on new models, we only commit the migrations at the very end when merging into master.

At last, we have the annual IBL meeting over the week-end so I wouldn't expect fast feedback until next Monday. There will be more topics to discuss if we continue (newsletters, separate apps) but I suggest we start small with those 2 features.

[bendichter]

@oliche perfect. We have already created a fork here: https://github.com/catalystneuro/alyx and we plan to make the changes there and PR (unless you would prefer we branch here). We have a meeting with the Buffalo Lab on Monday to dig into the specifics. We'll put something together over the next week.

Your help on the admin interface and REST endpoints/filters would be much appreciated! We have some members on our team that specialize in Django so we may be able to lift some of that load as well.

[oliche]

Ok as far as I'm concerned I just think we should be able to ping-pong commits on the same branch, wherever it is. On way is for me to give you access to the repo, another (I think) is to open an early pull-request from your fork (to dev) so I can push to it.

I'm fine with either.

[luiztauffer]

@oliche Here’s the tracking for the two proposed tables. Feel free to comment! We’ll have a meeting with the lab’s member next week and these will probably change/extend. https://github.com/catalystneuro/alyx/issues/3 https://github.com/catalystneuro/alyx/issues/1

As there’ll probably be different groups of people commiting code, perhaps doing it in our fork will be better?

[iandol]

It appears, sadly, that the attempt to extend Alyx to be less rodent-centric didn't go anywhere; they made many changes but never made any pull requests. Would current Alyx maintainers still consider making Alyx more generally flexible for other species and tasks?

[iandol]

Is there any useful information on extending Alyx, @oliche or @rossant or @kdharris101? Alyx could serve as a really useful tool for primate labs. Our Inst. (Centre for excellence in brain science & International centre for primate brain research, Chinese Academy of Sciences) needs a database specialised for primate research, we want to try to promote better open sharing of data (see https://www.digital-brain.cn for our current connectome stuff). I would rather contribute to Alyx than build our own as standard tools are better than endless recreations of the same thing. But we don't know if the Alyx maintainers are interested, we can fork and build to start with. I saw @k1o0 has a pull for some task changes, and I don't want to end up conflicting with your future plans.

[oliche]

Hello,

We tried to keep Alyx rather generic, but the project may not have been built with the requirements you have in mind ! One way is to contribute model changes to the main repository, in a way that keeps compatibility with our running instances. The fork is certainly the other option if you wish to modify the base models freely.

Yet there is an in-between solution in between absolute freedom and absolute collaboration. We faced the same issue at IBL of having very specific requirements and we have implemented those specific tools in an ibl_reports app so that we wouldn't push those onto every user's database. If you are not familiar, in Django jargon, an "app" is a new package containing models, views, migrations and templates. You can think of it as a plugin.

So you could create your own plugin of Alyx while keeping the base models common, as we did for our reports. When you feel base model changes are justified, we can go through the process of discussing proposed changes on the main apps.

If you want to see such an example in practice, this is where our app lives (in a separate repository): https://github.com/int-brain-lab/iblalyx/tree/main/management/ibl_reports

We then clone the repository in the same docker/machine running alyx and symlink the app into the alyx repo: https://github.com/int-brain-lab/iblalyx/blob/3718f82e379c7f02b53791c0273705511918485a/deploy/container/Dockerfile#L42

This requires modifying the .gitignore the settings.py and the urls.py files which we can happily do on the main branch to extend to third party apps. We have not documented the process, but this may be a good opportunity to do so if you choose to go this way.

PS: BTW the open data figuring on the website above looks amazing https://www.digital-brain.cn/ !

[iandol]

Thanks @oliche, that is helpful. I am still at the planning stage and I am not a Python person so I will be working with others for the low-level details. Are the relevant docs from Django this: https://docs.djangoproject.com/en/5.0/ref/applications/ ? How much can we change in terms of the schema etc.[1]? Given our inexperience with Django, we may find hacking on a fork easier to begin with, then use the experience to think about the "plugin" approach?

---

[1] we mostly want to add more structured details and flexibility for the task (where we will have many different task designs that need to be somehow structured), "hide" the things like colony management, and augment some other fields etc.