Closed dougollerenshaw closed 3 years ago
rly
commented
3 years ago This looks like an incompatibility between PyNWB and HDMF. PyNWB 1.5 which has the line causing the error requires HDMF >= 2.5.1. Can you please check what versions of PyNWB and HDMF are installed?
rly
commented
3 years ago Oh. I see that allensdk requires HDMF < 2.5 because there were issues with 2.5.0 and 2.5.1. https://github.com/AllenInstitute/AllenSDK/pull/2099
The raised issues should be fixed now in HDMF 2.5.2, so I think it would be safe to require pynwb>=1.5.1,<2 and hdmf>=2.5.2,<3.
dougollerenshaw
commented
3 years ago Here's the output of pip list in colab after calling !pip install allensdk
Package Version
----------------------------- -------------------
absl-py 0.12.0
aiohttp 3.7.4
alabaster 0.7.12
albumentations 0.1.12
allensdk 2.11.2
altair 4.1.0
appdirs 1.4.4
argon2-cffi 20.1.0
argschema 2.0.2
arviz 0.11.2
astor 0.8.1
astropy 4.2.1
astunparse 1.6.3
async-generator 1.10
async-timeout 3.0.1
atari-py 0.2.9
atomicwrites 1.4.0
attrs 21.2.0
audioread 2.1.9
autograd 1.3
Babel 2.9.1
backcall 0.2.0
beautifulsoup4 4.6.3
bleach 3.3.0
blis 0.4.1
bokeh 2.3.2
boto3 1.17.21
botocore 1.20.96
Bottleneck 1.3.2
branca 0.4.2
bs4 0.0.1
CacheControl 0.12.6
cached-property 1.5.2
cachetools 4.2.2
catalogue 1.0.0
certifi 2021.5.30
cffi 1.14.5
cftime 1.5.0
chardet 3.0.4
click 7.1.2
cloudpickle 1.3.0
cmake 3.12.0
cmdstanpy 0.9.5
colorcet 2.0.6
colorlover 0.3.0
community 1.0.0b1
contextlib2 0.5.5
convertdate 2.3.2
coverage 3.7.1
coveralls 0.5
crcmod 1.7
cufflinks 0.17.3
cvxopt 1.2.6
cvxpy 1.0.31
cycler 0.10.0
cymem 2.0.5
Cython 0.29.23
daft 0.0.4
dask 2.12.0
datascience 0.10.6
debugpy 1.0.0
decorator 4.4.2
defusedxml 0.7.1
descartes 1.1.0
dill 0.3.3
distributed 1.25.3
distro 1.5.0
dlib 19.18.0
dm-tree 0.1.6
docopt 0.6.2
docutils 0.17.1
dopamine-rl 1.0.5
earthengine-api 0.1.269
easydict 1.9
ecos 2.0.7.post1
editdistance 0.5.3
en-core-web-sm 2.2.5
entrypoints 0.3
ephem 3.7.7.1
et-xmlfile 1.1.0
fa2 0.3.5
fastai 1.0.61
fastdtw 0.3.4
fastprogress 1.0.0
fastrlock 0.6
fbprophet 0.7.1
feather-format 0.4.1
filelock 3.0.12
firebase-admin 4.4.0
fix-yahoo-finance 0.0.22
Flask 1.1.4
flatbuffers 1.12
folium 0.8.3
future 0.16.0
gast 0.4.0
GDAL 2.2.2
gdown 3.6.4
gensim 3.6.0
geographiclib 1.50
geopy 1.17.0
gin-config 0.4.0
glob2 0.7
Glymur 0.8.19
google 2.0.3
google-api-core 1.26.3
google-api-python-client 1.12.8
google-auth 1.31.0
google-auth-httplib2 0.0.4
google-auth-oauthlib 0.4.4
google-cloud-bigquery 1.21.0
google-cloud-bigquery-storage 1.1.0
google-cloud-core 1.0.3
google-cloud-datastore 1.8.0
google-cloud-firestore 1.7.0
google-cloud-language 1.2.0
google-cloud-storage 1.18.1
google-cloud-translate 1.5.0
google-colab 1.0.0
google-pasta 0.2.0
google-resumable-media 0.4.1
googleapis-common-protos 1.53.0
googledrivedownloader 0.4
graphviz 0.10.1
greenlet 1.1.0
grpcio 1.34.1
gspread 3.0.1
gspread-dataframe 3.0.8
gym 0.17.3
h5py 2.10.0
hdmf 2.4.0
HeapDict 1.0.1
hijri-converter 2.1.2
holidays 0.10.5.2
holoviews 1.14.4
html5lib 1.0.1
httpimport 0.5.18
httplib2 0.17.4
httplib2shim 0.0.3
humanize 0.5.1
hyperopt 0.1.2
ideep4py 2.0.0.post3
idna 2.10
imageio 2.4.1
imagesize 1.2.0
imbalanced-learn 0.4.3
imblearn 0.0
imgaug 0.2.9
importlib-metadata 4.5.0
importlib-resources 5.1.4
imutils 0.5.4
inflect 2.1.0
iniconfig 1.1.1
install 1.3.4
intel-openmp 2021.2.0
intervaltree 2.1.0
ipykernel 4.10.1
ipython 5.5.0
ipython-genutils 0.2.0
ipython-sql 0.3.9
ipywidgets 7.6.3
itsdangerous 1.1.0
jax 0.2.13
jaxlib 0.1.66+cuda110
jdcal 1.4.1
jedi 0.18.0
jieba 0.42.1
Jinja2 2.11.3
jmespath 0.10.0
joblib 1.0.1
jpeg4py 0.1.4
jsonschema 2.6.0
jupyter 1.0.0
jupyter-client 5.3.5
jupyter-console 5.2.0
jupyter-core 4.7.1
jupyterlab-pygments 0.1.2
jupyterlab-widgets 1.0.0
kaggle 1.5.12
kapre 0.3.5
Keras 2.4.3
keras-nightly 2.5.0.dev2021032900
Keras-Preprocessing 1.1.2
keras-vis 0.4.1
kiwisolver 1.3.1
korean-lunar-calendar 0.2.1
librosa 0.8.1
lightgbm 2.2.3
llvmlite 0.34.0
lmdb 0.99
LunarCalendar 0.0.9
lxml 4.2.6
Markdown 3.3.4
MarkupSafe 2.0.1
marshmallow 3.0.0rc6
matplotlib 3.2.2
matplotlib-inline 0.1.2
matplotlib-venn 0.11.6
missingno 0.4.2
mistune 0.8.4
mizani 0.6.0
mkl 2019.0
mlxtend 0.14.0
more-itertools 8.8.0
moviepy 0.2.3.5
mpmath 1.2.1
msgpack 1.0.2
multidict 5.1.0
multiprocess 0.70.11.1
multitasking 0.0.9
murmurhash 1.0.5
music21 5.5.0
natsort 5.5.0
nbclient 0.5.3
nbconvert 5.6.1
nbformat 5.1.3
ndx-events 0.2.0
nest-asyncio 1.2.0
netCDF4 1.5.6
networkx 2.5.1
nibabel 3.0.2
nltk 3.2.5
notebook 5.3.1
numba 0.51.2
numexpr 2.7.3
numpy 1.18.5
nvidia-ml-py3 7.352.0
oauth2client 4.1.3
oauthlib 3.1.1
okgrade 0.4.3
opencv-contrib-python 4.1.2.30
opencv-python 4.1.2.30
openpyxl 2.5.9
opt-einsum 3.3.0
osqp 0.6.2.post0
packaging 20.9
palettable 3.3.0
pandas 0.25.3
pandas-datareader 0.9.0
pandas-gbq 0.13.3
pandas-profiling 1.4.1
pandocfilters 1.4.3
panel 0.11.3
param 1.10.1
parso 0.8.2
pathlib 1.0.1
patsy 0.5.1
pexpect 4.8.0
pickleshare 0.7.5
Pillow 7.1.2
pip 19.3.1
pip-tools 4.5.1
plac 1.1.3
plotly 4.4.1
plotnine 0.6.0
pluggy 0.7.1
pooch 1.4.0
portpicker 1.3.9
prefetch-generator 1.0.1
preshed 3.0.5
prettytable 2.1.0
progressbar2 3.38.0
prometheus-client 0.11.0
promise 2.3
prompt-toolkit 1.0.18
protobuf 3.12.4
psutil 5.4.8
psycopg2 2.7.6.1
psycopg2-binary 2.9
ptyprocess 0.7.0
py 1.10.0
pyarrow 3.0.0
pyasn1 0.4.8
pyasn1-modules 0.2.8
pycocotools 2.0.2
pycparser 2.20
pyct 0.4.8
pydata-google-auth 1.2.0
pydot 1.3.0
pydot-ng 2.0.0
pydotplus 2.0.2
PyDrive 1.3.1
pyemd 0.5.1
pyerfa 2.0.0
pyglet 1.5.0
Pygments 2.6.1
pygobject 3.26.1
pymc3 3.11.2
PyMeeus 0.5.11
pymongo 3.11.4
pymystem3 0.2.0
pynrrd 0.4.2
pynwb 1.5.1
PyOpenGL 3.1.5
pyparsing 2.4.7
pyrsistent 0.17.3
pysndfile 1.3.8
PySocks 1.7.1
pystan 2.19.1.1
pytest 3.6.4
python-apt 0.0.0
python-chess 0.23.11
python-dateutil 2.8.1
python-louvain 0.15
python-slugify 5.0.2
python-utils 2.5.6
pytz 2018.9
pyviz-comms 2.0.2
PyWavelets 1.1.1
PyYAML 3.13
pyzmq 22.1.0
qdldl 0.1.5.post0
qtconsole 5.1.0
QtPy 1.9.0
regex 2019.12.20
requests 2.23.0
requests-oauthlib 1.3.0
requests-toolbelt 0.9.1
resampy 0.2.2
retrying 1.3.3
rpy2 3.4.5
rsa 4.7.2
ruamel.yaml 0.17.9
ruamel.yaml.clib 0.2.2
s3transfer 0.3.7
scikit-build 0.11.1
scikit-image 0.16.2
scikit-learn 0.22.2.post1
scipy 1.4.1
screen-resolution-extra 0.0.0
scs 2.1.4
seaborn 0.11.1
semver 2.13.0
Send2Trash 1.5.0
setuptools 57.0.0
setuptools-git 1.2
Shapely 1.7.1
simplegeneric 0.8.1
SimpleITK 2.0.2
simplejson 3.17.2
six 1.15.0
sklearn 0.0
sklearn-pandas 1.8.0
smart-open 5.1.0
snowballstemmer 2.1.0
sortedcontainers 2.4.0
SoundFile 0.10.3.post1
spacy 2.2.4
Sphinx 1.8.5
sphinxcontrib-serializinghtml 1.1.5
sphinxcontrib-websupport 1.2.4
SQLAlchemy 1.4.18
sqlparse 0.4.1
srsly 1.0.5
statsmodels 0.10.2
sympy 1.7.1
tables 3.6.1
tabulate 0.8.9
tblib 1.7.0
tensorboard 2.5.0
tensorboard-data-server 0.6.1
tensorboard-plugin-wit 1.8.0
tensorflow 2.5.0
tensorflow-datasets 4.0.1
tensorflow-estimator 2.5.0
tensorflow-gcs-config 2.5.0
tensorflow-hub 0.12.0
tensorflow-metadata 1.0.0
tensorflow-probability 0.12.1
termcolor 1.1.0
terminado 0.10.1
testpath 0.5.0
text-unidecode 1.3
textblob 0.15.3
Theano-PyMC 1.1.2
thinc 7.4.0
tifffile 2021.6.14
toml 0.10.2
toolz 0.11.1
torch 1.8.1+cu101
torchsummary 1.5.1
torchtext 0.9.1
torchvision 0.9.1+cu101
tornado 5.1.1
tqdm 4.41.1
traitlets 5.0.5
tweepy 3.10.0
typeguard 2.7.1
typing-extensions 3.7.4.3
tzlocal 1.5.1
uritemplate 3.0.1
urllib3 1.24.3
vega-datasets 0.9.0
wasabi 0.8.2
wcwidth 0.2.5
webencodings 0.5.1
Werkzeug 1.0.1
wheel 0.36.2
widgetsnbextension 3.5.1
wordcloud 1.5.0
wrapt 1.12.1
xarray 0.15.1
xgboost 0.90
xkit 0.0.0
xlrd 1.1.0
xlwt 1.3.0
yarl 1.6.3
yellowbrick 0.9.1
zict 2.0.0
zipp 3.4.1 @rly and @djkapner it looks like upgrading hdmf in colab gets me around this issue. In the first cell, I call
!pip install allensdk
!pip install hdmf --upgradeThen in a subsequent cell I can call
import allensdk.brain_observatory.behavior.behavior_project_cache as bpcwithout error.
rly
commented
3 years ago Glad that upgrading hdmf worked.
For further context, your colab environment is using an older version of pip. pip < 20.2 allowed creating incompatible installations which is what happened here. I recommend upgrading the version of pip on colab before running pip install so that this error is caught and fixed during pip install. (If you just use the latest version of pip and do not change the allensdk requirements or upgrade hdmf, then pip will install older versions of pynwb and hdmf that are compatible with each other.)
djkapner
commented
3 years ago @dougollerenshaw
I am running tests taking Ryan's suggestions for the new requirements limits. It'll take several hours to work it's way through.
In the meantime, you could check in colab with something like:
pip install git+https://github.com/AllenInstitute/AllenSDK@bugfix/nwb_dependency
also possible this change breaks something else for us, so, it is worth us waiting to let all the tests run.
dougollerenshaw
commented
3 years ago Thanks @rly, you're right. I put this in the top cell in colab:
!python -m pip install --upgrade pipThen
!pip install allensdkI get a warning about needing to restart my runtime that looks like this:
After clicking RESTART RUNTIME, I can then run:
import allensdk.brain_observatory.behavior.behavior_project_cache as bpcwithout error.
dougollerenshaw
commented
3 years ago @djkapner running
!pip install git+https://github.com/AllenInstitute/AllenSDK@bugfix/nwb_dependencyin the first cell gives the same behavior as in my last comment. As long as I click the RESTART RUNTIME button, the bpc import works.
djkapner
commented
3 years ago It will be a little longer. I see this is pushing up the requirements for numpy and pandas. I think we've done the work for this and were waiting for these hdmf and pynwb releases, but, there's opportunity for finding more gotchas.
wbwakeman
commented
3 years ago Closing this issue because we believe the 'colab' issue was resolved. We will address the pynwb/hdmf version issue, and resulting dependencies and failing tests in a different issue.
Describe the bug When trying to set up a google colab notebook to use the SDK, I get the error:
To Reproduce In cell 1 of a colab notebook, run:
Then in cell 2, run:
Here is a link to a colab notebook demonstrating: https://colab.research.google.com/drive/1m2sMgWw3K2DYAUXn1COo58RAELUkBh00?usp=sharing
Expected behavior I expect the above import to run without error.
Actual Behavior On first run, there is a pandas compatibility issue. The allensdk is pinned at 0.25.3, but colab runs 1.1.5. This can be circumvented by restarting the runtime, which allows the pinned version to override the default
But, critically, after restarting the runtime and re-executing, we get this:
Or, in text form (with all frames expanded):
Environment (please complete the following information):
Additional context None
Do you want to work on this issue? Yes, I'd like to help troubleshoot