Implementing two new models for BlueNaaS

[alex4200]

As a request of @SteMasoli he wants his two models be implemented into a list of models to be selected in the BSP (for BlueNaaS):

Single section model. https://blue-naas.humanbrainproject.eu/#/model/grc_mono Temperature = 30 Vinit = 80 Current clamp = from 0.01 to 0.03nA

Multi section model https://blue-naas.humanbrainproject.eu/#/model/grc_multi Temperature = 30 Vinit = 70 Current clamp = from 0.01 to 0.03nA

The two models could be added to the list of models (Luca is handling so far I know)

OR we could insert a new selection (as it is done when selecting the "Small Circuit In Silico Experiments" use case, but naming instead e.g. "Please select a region" and the you can select 'hippocampus' and 'granule cells' or something):

[SteMasoli]

@genric I wrote to Luca about the way they use to upload the models. Have I to create a new specific container? The models, to be uploaded, will be a dozen or more.

[clupascu]

@genric Where are you fetching the list of models to be selected in the BlueNaaS from? Are you still using https://github.com/lbologna/bsp_data_repository/?

[genric]

I am currently trying to make hippocampus models available as MINDS dataset over here: https://www.humanbrainproject.eu/en/explore-the-brain/search/?facet_type[0]=Dataset I think this is how all models will be published in the future: using HBP knowledge graph(NEXUS). If you want, we could try integrating your models as well. For that I would need the set of zip files(which you tested with BlueNaaS) or a git repository url with all your models you wish to publish. Have in mind that this integration might take some time(weeks probably). So if you need this urgently we could just extend the existing list and do it the old way.

[jdcourcol]

@genric be careful as @clupascu data are 'optimization' not just single cell model.

[genric]

@clupascu Yes, they were fetched from that repository. And the ones I'll be uploading to HBP NEXUS I'll fetch from there.

[genric]

@jdcourcol Yes, in the first iteration single cell models will contain only necessary information to load them to BlueNaaS. Later we can attach more metadata regarding optimization. They will contain the morphology which was used to run the optimization because the same morphology will be used in BlueNaaS.

[clupascu]

@genric As GitHub recommends repositories to be kept under 1GB each and our repository is already at 800Mb, we want to add new models to the CSCS container that was created for the optimizations https://object.cscs.ch/v1/AUTH_c0a333ecf7c045809321ce9d9ecdfdea/hippocampus_optimization/ (even if there is no versioning here). Can you fetch the models from a container too?

[genric]

@clupascu Yep, I have the access. Actually this container can be attached to the knowledge graph dataset as raw data. So people who wish to access optimization metadata can follow the link.

[apdavison]

@genric Please note that making models available through the HBP Knowledge Graph is something we are also working on in my team, as part of our work in SP5. We are currently migrating the Model Catalog to the Knowledge Graph, so that in future there will be two ways to access information about models: through the Model Catalog API and directly through the KG APIs.

We should coordinate our work on this to avoid duplication of effort.

For the time being, even before completing the KG integration, the list of models could (should, I think) be obtained through the Model Catalog API - see https://hbp-validation-framework-api.readthedocs.io/en/latest/models.html#list-model-descriptions-that-satisfy-specified-filters and let us know if the filters we have are inadequate.

[clupascu]

@apdavison Are you fetching the models in the Model Catalog from github too?

[apdavison]

@clupascu We do not automatically update the Model Catalog from github, but I understood that all models from the github repository were imported into the MC, and that you were no longer using github for this purpose. @appukuttan-shailesh can you comment further?

(Maybe we could all work on this together at the CodeJam in a couple of weeks. @genric will you be going to the CodeJam?)

[clupascu]

@apdavison we would like to no longer use github for this purpose and upload new models to cscs containers directly, if everybody is convinced to use the containers only to fetch the models. We can work on this together at the CodeJam.

[genric]

@apdavison yes, I'll be in the CodeJam. And my idea about this HBP KG initial model integration was to have something for the CodeJam to "jam" about :) even if it uses the outdated models from the github.

[apdavison]

@clupascu ok, I thought "models-at-CSCS" was how you had been working since before the end of SGA1

@genric by using the Model Catalog API to get a list of models, the transition to KG should be seamless, since at some point the API will just switch from using its own backend database to using the KG as backend.

[appukuttan-shailesh]

@apdavison : Yes as part of end of SGA1 activities, and subsequently for the live-paper, we had registered all (I believe we covered ALL) models from the CSCS containers on the model catalog. We have stopped keeping track of the GitHib repository now (unless having specific requests for the same).

@genric : We had discussed some weeks ago about integrating BlueNaaS with the model catalog. At the time, we had proposed taking this up at the CodeJam later this month. Hope we can get atleast something preliminary working.

[genric]

@apdavison I see, thanks for the update!

[genric]

@appukuttan-shailesh yeah, and recently I've learned about these MINDS datasets, so I was trying to make the models indexed and available in the HBP KG search. At the CodeJam there will be also people from the HBP KG team, so looks like it will be a pretty busy CodeJam :) looking forward to it!

[SteMasoli]

I hope there will be some documentation for those, like me, that will not attend the CodeJam.

[appukuttan-shailesh]

Moving the discussion back to here (from #351):

@SteMasoli : I saw the google doc.... so I suppose we are referring to rows 60, 61 and 62. I have verified that the models are publicly accessible and in BlueNaaS appropriate format. I shall upload these three models to the model catalog and let you know.

Regarding (from #351):

The file already on the mode catalog need to be deleted.

I don't think we need to delete these. The model catalog is intended to retain old data, and not just the latest models. Do let us know if there are other reasons for wanting to delete these.

@alex4200 : Somebody needs to handle the adding of these models to the 'UseCase' list (see Figs. 3 and 4 in first post). Not sure who handles this.

[SteMasoli]

@appukuttan-shailesh

I shall upload these three models to the model catalog and let you know.

Thank you.

Do let us know if there are other reasons for wanting to delete these.

To avoid misunderstandings when an user will search for a model, in the model catalog, and will find the single files instead of the packages for BlueNaaS.

Somebody needs to handle the adding of these models to the 'UseCase' list.

This can be work around for the previous comment. Having the BlueNaaS models accessible from the BSP will solve the problem of searching in the model catalog and will let me define the appropriate parameters to do the simulations.

[appukuttan-shailesh]

@SteMasoli : The three models have been uploaded to the model catalog, you can find them here:

1) Granule cell - Mono compartmental https://collab.humanbrainproject.eu/#/collab/9136/nav/77077?state=model.527191ca-8385-4aac-b08c-3bce022fc5cc

2) Granule cell - Multi compartmental https://collab.humanbrainproject.eu/#/collab/9136/nav/77077?state=model.0c97cd83-24fd-4782-bdea-2affacc6863e

3) Purkinje cell - Multi compartmental https://collab.humanbrainproject.eu/#/collab/9136/nav/77077?state=model.d3a1a6f7-d4c6-4e4a-b1d8-76e470f20862

Do take a look and see if they are ok. If not, you can edit the entries via the above links (click the "Edit" button).

I haven't deleted the older entries as they seemed to be different from the current entries (e.g. the species was 'Rat' for some of the older entries). We could possibly decide on the deletion later.

@genric : As replied via email, we are now able to use public CSCS storage container URLs to launch models in BlueNaaS. Thanks for getting this to work. The model catalog is expected to have this BlueNaaS feature implemented in around 2 weeks (you can track progress here: https://github.com/HumanBrainProject/hbp-validation-framework/issues/228).

@alex4200 :

Somebody needs to handle the adding of these models to the 'UseCase' list (see Figs. 3 and 4 in first post). Not sure who handles this.

Might be good to have this sub-task assigned to someone so that it isn't overlooked.

[antonelepfl]

Somebody needs to handle the adding of these models to the 'UseCase' list (see Figs. 3 and 4 in first post). Not sure who handles this.

I can take care of that but I need information about how to access the Model Catalog API as I asked on the Monday meeting.

[appukuttan-shailesh]

@antonelepfl : Here is the docs for the Model Catalog REST API: https://hbp-validation-framework-api.readthedocs.io

We also have a Python client if needed: https://hbp-validation-client.readthedocs.io

[antonelepfl]

@genric @appukuttan-shailesh I was checking the content of the models and I saw there is a morphology picture but I don't find the traces for each model as image. In Github we had the traces images + the morphologies (Fig 3). Is it somewhere else than associated with the models?

For example the detailed view for https://validation-v1.brainsimulation.eu/models/?id=8f4f2800-032e-4e4e-9558-0062bd347b47

[appukuttan-shailesh]

The traces were not uploaded to the model entries previously. Just last week @clupascu proposed that we could upload these as well. We shall ensure that this is updated for future model entries, and for the models relevant to the live paper.

Do you have a specific use for these images? Currently they are provided on the model catalog entries mostly as additional info.

[antonelepfl]

Specific use I think is the model list that I present before selecting a use case (Fig 3)

[appukuttan-shailesh]

How do you intend to shortlist the models to be displayed for the use case (what filters)? I suppose it won't be feasible (or appropriate?) to display every entry from the model catalog as these would be numerous.

[appukuttan-shailesh]

@clupascu, @antonelepfl Update: The responses image has now been uploaded for all the models relevant to the live paper.

[antonelepfl]

Cool thank you @appukuttan-shailesh

[antonelepfl]

Is there any remain topic here? Can we close this ticket? BTW the models are in dev version #360

[alex4200]

For me its ok to close this ticket...