TVB Jupyter Notebooks in the BSP Online Use Cases

lbologna commented 4 years ago

TVB Use Cases

Aspect	Detail
Summary	Three TVB Jupyter Notebooks to be potentially integrated in the BSP Online Use Cases
Usecase Group	Brain Area Circuit and/or Whole Brain In Silico Exp (new): to be decided after exploring the jupyter notebooks
Expert	@alex4200 @liadomide
Scientific User	@alex4200, @lbologna
Deadline	February 2020
Target audience	Scientific Community
Target interface
HPC Requirements
Dependencies
Nominal runtime

List of features

The new use cases allow to run TVB simulations via Jupyter Notebooks of the BSP Online Use Cases. I report below the original indications from Lia Diomide (Codemart/TVB): """ As discussed, I have selected some notebooks which might be useful to be shown under BSP -> Online Use Cases, as Notebook examples for using TVB software in scientifically relevant use-cases: A - "Simulate whole brain activity with reduced Wong Wang model from TVB" https://collab.humanbrainproject.eu/#/collab/1609/nav/15656 B - "Simulate and apply stimulus in TVB" https://collab.humanbrainproject.eu/#/collab/1609/nav/15655 C - "Simulate for mouse, with TVB models and Allen DB driven data" https://collab.humanbrainproject.eu/#/collab/1609/nav/27162

These notebooks rely on input data from here: https://collab.humanbrainproject.eu/#/collab/1609/nav/14242 tvb_data/connectivity/connectivity_76.zip tvb_data/mouse/allen_2mm """

Tasks

[ ] Test Notebook 1 (DEV) with indication on changes and integrations to the developers
[ ] Test Notebook 2 (DEV) with indication on changes and integrations to the developers
[ ] Test Notebook 3 (DEV) with indication on changes and integrations to the developers
[ ] Build BSP Guidebook pages for Jupyter Notebook 1
[ ] Build BSP Guidebook pages for Jupyter Notebook 2
[ ] Build BSP Guidebook pages for Jupyter Notebook 3
[ ] Check BSP Guidebook pages for Jupyter Notebook 1
[ ] Check BSP Guidebook pages for Jupyter Notebook 2
[ ] Check BSP Guidebook pages for Jupyter Notebook 3
[ ] Move Jupyter Notebook 1 to production
[ ] Move Jupyter Notebook 2 to production
[ ] Move Jupyter Notebook 3 to production

Acceptance Criteria

All Jupyter Notebooks work well and the related Guidebook pages provides complete information on which the notebooks' goals are and how to use them

alex4200 commented 4 years ago

@liadomide Hi Lia, I see that you are using the following function in the use cases:

def download_file(file_name):
    client = get_bbp_client().document
    COLLAB_PATH = get_collab_storage_path()
    client.listdir(COLLAB_PATH)
    collab_full_path = join(COLLAB_PATH, 'tvb_data', "connectivity", file_name)
    client.download_file(collab_full_path, file_name)

As get_bbp_client() is outdated, please replace it with the following code in all of the notebooks:

def download_file(file_name):
    client = get_hbp_service_client()
    COLLAB_PATH = "/TestingNotebook"    
    collab_full_path = os.path.join(COLLAB_PATH, 'tvb_data', "connectivity", file_name)
    client.storage.download_file(collab_full_path, file_name)

Please give it a try.

In case you get an error please check #495.

liadomide commented 4 years ago

@alex4200 We did try the new code, but method download_file does not appear to exist. We found download_file_content instead, but it is a bit unclear how exactly to use it, as it requires a GUID. Could it happen that you have some documentation on this new library API?

alex4200 commented 4 years ago

@liadomide You have to make sure that you have hbp-service-client installed with at least version 1.0.0.

Documentation can be found here: https://collab.humanbrainproject.eu/#/collab/509/nav/41653

gabiflorea10 commented 4 years ago

@alex4200 We make sure that we have hbp-service-client installed with last version (1.1.1). But this does not solve the problem. The method download_file still not appear to exist. I attach two relevant screenshots. collab_error error_hbp_collab_2

alex4200 commented 4 years ago

@gabiflorea10 Can you please try the following:

Make sure you use a python3 kernel
Restart the kernel

and then try again

gabiflorea10 commented 4 years ago

It works now. Thank you for support.

alex4200 commented 4 years ago

@gabiflorea10 Let me know when you have updated the three notebooks so I can have a look again.

gabiflorea10 commented 4 years ago

@alex4200 For the moment I have some problems in running notebooks on collab. One of them is in notebook: "Run a custom cosimulation" from TVB NEST collab. After running a job, when I try to get results, I run into an error:

HTTPError Traceback (most recent call last)

in () ----> 1 out = wd.stat("stdout") 2 f = out.raw() 3 4 all_lines = f.read().splitlines() 5 all_lines[-20:] /opt/conda/lib/python3.6/site-packages/pyunicore/client.py in stat(self, path) 405 path_url = self.path_urls['files'] + '/' + path 406 headers = {'Accept': 'application/json',} --> 407 props = self.transport.get(url=path_url, headers = headers) 408 if props['isDirectory']: 409 ret = PathDir(self, path_url, path) /opt/conda/lib/python3.6/site-packages/pyunicore/client.py in get(self, to_json, **kwargs) 168 headers = self._headers(kwargs) 169 req = requests.get(headers=headers, verify=self.verify, **kwargs) --> 170 req.raise_for_status() 171 if not to_json: 172 return req /opt/conda/lib/python3.6/site-packages/requests/models.py in raise_for_status(self) 938 939 if http_error_msg: --> 940 raise HTTPError(http_error_msg, response=self) 941 942 def close(self): HTTPError: 404 Client Error: Not Found for url: https://brissago.cscs.ch:8080/DAINT-CSCS/rest/core/storages/b6491ad5-798f-45c9-a642-3ea30bf6ab25-uspace/files/stdout How you suggest to deal with this error?

alex4200 commented 4 years ago

@gabiflorea10 Can you log in to PizDaint and check the content of the folder yourself? I suspect there is no file stdout.

I tried to check but I cannot access the folder /scratch/snx3000/unicore/FILESPACE/b6491ad5-798f-45c9-a642-3ea30bf6ab25/' on PizDaint (Permission denied`).

gabiflorea10 commented 4 years ago

@alex4200 I checked what you required and I see a file named 'stdout' in folder you indicate to search in. I also tried, just for testing purposes, to change the permissions for stdout to '777'. I still have the same error.

gabiflorea10 commented 4 years ago

@alex4200 I have one more problem in running notebooks in collab, error that is possible to be related to the aforementioned one. In TVB NEST collab, in notebook 'Test TVB-NEST installation', I use a '.sh' file to specify the simulation I want to perform. This file has the following content: I can run this '.sh' file directly from PuTTy and it work as intended, but when I try to run it in collab, I run into the following error: Have you any suggestions?

alex4200 commented 4 years ago

@gabiflorea10 I am not sure what it the goal of running the script. Do you want to run something on a computer cluster from the notebook? And what is saurus?

Maybe we can have a short voice/video call this week?

alex4200 commented 4 years ago

Still no update on this issue

antonelepfl commented 4 years ago

Still no update on this issue

liadomide commented 4 years ago

We managed to have our notebooks running in our Collabs, so everything fine there. Although, the initial description of this task is a bit different. Not sure what the status is for that part (publish differently)

alex4200 commented 4 years ago

@liadomide Can you remind me again where to find your DEV notebooks? Are they still located at those locations?

A - "Simulate whole brain activity with reduced Wong Wang model from TVB" https://collab.humanbrainproject.eu/#/collab/1609/nav/15656 B - "Simulate and apply stimulus in TVB" https://collab.humanbrainproject.eu/#/collab/1609/nav/15655 C - "Simulate for mouse, with TVB models and Allen DB driven data" https://collab.humanbrainproject.eu/#/collab/1609/nav/27162

If so, then none of these notebooks work for me, every notebook creates an error in the first cell.

Maybe I have to run something else before? Maybe I require some special setup/environment?

If you like we can also have a skype call to discuss the notebooks and errors.

liadomide commented 4 years ago

Yes, these should be the link. What is your error exactly? Did you copy the notebooks in your own space? Have you also copied the data from storage?

alex4200 commented 4 years ago

Hi @liadomide. I did copy the notebooks to a new Collab and executed them there. I did not yet copy the data from the storage (you can either download the required files from the original Collab 1609, or you can pack all the data into a zip file which will be downloaded and unpacked in a notebook).

So for the first two notebooks A&B the first cell gives an error:

Populating the interactive namespace from numpy and matplotlib
Traceback (most recent call last):

  File "/opt/conda/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2862, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)

  File "<ipython-input-1-d9540457154d>", line 3, in <module>
    from tvb.simulator.lab import *

  File "/opt/conda/lib/python3.6/site-packages/tvb/simulator/lab.py", line 39, in <module>
    from tvb.basic.profile import TvbProfile

  File "/opt/conda/lib/python3.6/site-packages/tvb/basic/profile.py", line 45, in <module>
    from tvb.basic.config.profile_settings import BaseSettingsProfile

  File "/opt/conda/lib/python3.6/site-packages/tvb/basic/config/profile_settings.py", line 54
    ACCESS_MODE_TVB_FILES = 0744
                               ^
SyntaxError: invalid token

For notebook C I strongly suggest to change the first cell to the following code:

!pip install -q -I --upgrade hbp_service_client
from os.path import join

def download_file(file_name):
    client = get_hbp_service_client().storage
    COLLAB_PATH = "/TestingNotebook"
    client.list(COLLAB_PATH)
    collab_full_path = join(COLLAB_PATH, 'tvb_data', "mouse", "allen_2mm", file_name)
    client.download_file(collab_full_path, file_name)
    !ls

download_file("Connectivity.zip")

This will use the hbp_service_client (the bbp one is totally deprecated). But then I get an error soon when the code is trying to read the existing file Connectivity.zip:


IOErrorTraceback (most recent call last)
<ipython-input-3-ad13d5441d70> in <module>()
      1 # Load the connectivity data from the directory tvb_data
----> 2 con = connectivity.Connectivity.from_file("Connectivity.zip")
      3 
      4 # Visualize the structural connectivity matrix
      5 plt.subplots()

/opt/conda/envs/python2/lib/python2.7/site-packages/tvb/datatypes/connectivity.pyc in from_file(source_file, instance)
    932 
    933         else:
--> 934             reader = ZipReader(source_full_path)
    935 
    936             result.weights = reader.read_array_from_file("weights")

/opt/conda/envs/python2/lib/python2.7/site-packages/tvb/basic/readers.pyc in __init__(self, zip_path)
    162 
    163         self.logger = get_logger(__name__)
--> 164         self.zip_archive = zipfile.ZipFile(zip_path)
    165 
    166     def has_file_like(self, file_name):

/opt/conda/envs/python2/lib/python2.7/zipfile.pyc in __init__(self, file, mode, compression, allowZip64)
    754             modeDict = {'r' : 'rb', 'w': 'wb', 'a' : 'r+b'}
    755             try:
--> 756                 self.fp = open(file, modeDict[mode])
    757             except IOError:
    758                 if mode == 'a':

IOError: [Errno 2] No such file or directory: '/opt/conda/envs/python2/lib/python2.7/site-packages/tvb_data/connectivity/Connectivity.zip'

That error points to a deeper problem, as the file exists in the current folder, but the code is trying to read the file from some strange folder '/opt/conda/envs/python2/lib/python2.7/site-packages/tvb_data/connectivity/.

In general a basic suggestion: Please test your notebook at least once before you give it to me for testing!

liadomide commented 4 years ago

Hi @alex4200,

Apologies as these are not working for you. We did test them, but apparently not in a clean env.

First problem you encounter, SyntaxError: invalid token, it is caused by a recent change done in HBP Collab. A very old version of TVB is installed now, which is not even Python 3 compatible. Thus every notebook from us now has to have at the top: ! pip uninstall -y tvb ! pip install --upgrade tvb-library hbp-service-client We realized that not all of your notebooks had this at the top, although they should. We hope this is fixed now.
The second problem, regarding Connectivity.zip should be solved if you have in your Collab space, the mentioned connectivity under tvb_data/connectivity folder.
and Notebook C, is indeed the only one left not translated to the new HBP API, nor to TVB latest version, as there is still a bug in TVB on that section.

Thus, can you work with only the first 2 examples, or any other example from HBP Collab https://collab.humanbrainproject.eu/#/collab/1609

Nevertheless, I am welcoming a call, as it might save both of us valuable time.

liadomide commented 4 years ago

my skype ID is lia.domide

liadomide commented 4 years ago

After a call with @alex4200 we came to the following proposal of next steps:

we (codemart) will migrate at least one of these notebook to the new Collab
- then share it again, for check and integration
have one use-case in the new simulation platform where the TVB web GUI (deployed on HBP Open Shift stack) will be linked.
- To be defined how that will be documented/described, in addition to having the link towards the deployed app (this is the dev very first draft http://tvb-gui-route-tvb.apps-dev.hbp.eu/user/)

cnr-ibf-pa / hbp-bsp-issues