search

dattalab / moseq2-pca

Code for computing pcs from extracted data to submit for modeling
Other
0 stars 1 forks source link

Apply pca: Unable to copy object (destination object already exists) #51

Closed YudiXie closed 5 years ago

YudiXie commented 5 years ago

I always have the following error when running apply pca in my current dataset. I just updated moseq so I don't know if this is a problem with my dataset or it is because of my moseq installation. Previous analysis was fine.

Could anyone tell me what's wrong? Thanks in advance.

(moseq2) alex@UchidaGPU:/media/alex/DataDrive1/MoSeqData/7day_preexposure_MoSeq$ moseq2-pca apply-pca -c 6 -n 1
Loading PCs from /media/alex/DataDrive1/MoSeqData/7day_preexposure_MoSeq/_pca/pca.h5
/home/alex/miniconda3/envs/moseq2/lib/python3.6/site-packages/h5py/_hl/dataset.py:313: H5pyDeprecationWarning: dataset.value has been deprecated. Use dataset[()] instead.
  "Use dataset[()] instead.", H5pyDeprecationWarning)
Web UI served at 140.247.90.62/803/1:8787 (if port forwarding use internal IP not localhost)
Tunnel command:
 ssh -NL 8787:140.247.90.62/803/1:8787 UchidaGPU
Tunnel command (gcloud):
 gcloud compute ssh UchidaGPU -- -NL 8787:140.247.90.62/803/1:8787
Computing scores (batch 1/84):   0%|                      | 0/1 [00:00<?, ?it/s]/home/alex/miniconda3/envs/moseq2/lib/python3.6/site-packages/h5py/_hl/dataset.py:313: H5pyDeprecationWarning: dataset.value has been deprecated. Use dataset[()] instead.
  "Use dataset[()] instead.", H5pyDeprecationWarning)
Computing scores (batch 1/84): 100%|██████████████| 1/1 [00:14<00:00, 14.31s/it]
Computing scores (batch 2/84):   0%|                      | 0/1 [00:00<?, ?it/s]/home/alex/miniconda3/envs/moseq2/lib/python3.6/site-packages/h5py/_hl/dataset.py:313: H5pyDeprecationWarning: dataset.value has been deprecated. Use dataset[()] instead.
  "Use dataset[()] instead.", H5pyDeprecationWarning)
Traceback (most recent call last):
  File "/home/alex/miniconda3/envs/moseq2/bin/moseq2-pca", line 10, in <module>
    sys.exit(cli())
  File "/home/alex/miniconda3/envs/moseq2/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/alex/miniconda3/envs/moseq2/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/alex/miniconda3/envs/moseq2/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/alex/miniconda3/envs/moseq2/lib/python3.6/site-packages/moseq2_pca/util.py", line 49, in invoke
    return super(custom_command_class, self).invoke(ctx)
  File "/home/alex/miniconda3/envs/moseq2/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/alex/miniconda3/envs/moseq2/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/alex/miniconda3/envs/moseq2/lib/python3.6/site-packages/moseq2_pca/cli.py", line 347, in apply_pca
    mask_params=mask_params)
  File "/home/alex/miniconda3/envs/moseq2/lib/python3.6/site-packages/moseq2_pca/pca/util.py", line 284, in apply_pca_dask
    f.copy('/metadata/extraction', f_scores, name=metadata_name)
  File "/home/alex/miniconda3/envs/moseq2/lib/python3.6/site-packages/h5py/_hl/group.py", line 493, in copy
    copypl, base.dlcpl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5o.pyx", line 217, in h5py.h5o.copy
RuntimeError: Unable to copy object (destination object already exists)
jmarkow commented 5 years ago

Are you running the latest version of moseq2-pca? On Thu, Feb 28, 2019 at 8:55 AM Yu Xie notifications@github.com wrote:

I always have the following error when running apply pca in my current dataset. I just updated moseq so I don't know if this is a problem with my dataset or my moseq installation.

Could anyone tell me what's wrong? Thanks in advance.

(moseq2) alex@UchidaGPU:/media/alex/DataDrive1/MoSeqData/7day_preexposure_MoSeq$ moseq2-pca apply-pca -c 6 -n 1

Loading PCs from /media/alex/DataDrive1/MoSeqData/7day_preexposure_MoSeq/_pca/pca.h5

/home/alex/miniconda3/envs/moseq2/lib/python3.6/site-packages/h5py/_hl/dataset.py:313: H5pyDeprecationWarning: dataset.value has been deprecated. Use dataset[()] instead.

"Use dataset[()] instead.", H5pyDeprecationWarning)

Web UI served at 140.247.90.62/803/1:8787 (if port forwarding use internal IP not localhost)

Tunnel command:

ssh -NL 8787:140.247.90.62/803/1:8787 UchidaGPU

Tunnel command (gcloud):

gcloud compute ssh UchidaGPU -- -NL 8787:140.247.90.62/803/1:8787

Computing scores (batch 1/84): 0%| | 0/1 [00:00<?, ?it/s]/home/alex/miniconda3/envs/moseq2/lib/python3.6/site-packages/h5py/_hl/dataset.py:313: H5pyDeprecationWarning: dataset.value has been deprecated. Use dataset[()] instead.

"Use dataset[()] instead.", H5pyDeprecationWarning)

Computing scores (batch 1/84): 100%|██████████████| 1/1 [00:14<00:00, 14.31s/it]

Computing scores (batch 2/84): 0%| | 0/1 [00:00<?, ?it/s]/home/alex/miniconda3/envs/moseq2/lib/python3.6/site-packages/h5py/_hl/dataset.py:313: H5pyDeprecationWarning: dataset.value has been deprecated. Use dataset[()] instead.

"Use dataset[()] instead.", H5pyDeprecationWarning)

Traceback (most recent call last):

File "/home/alex/miniconda3/envs/moseq2/bin/moseq2-pca", line 10, in 

sys.exit(cli())

File "/home/alex/miniconda3/envs/moseq2/lib/python3.6/site-packages/click/core.py", line 764, in call

return self.main(*args, **kwargs)

File "/home/alex/miniconda3/envs/moseq2/lib/python3.6/site-packages/click/core.py", line 717, in main

rv = self.invoke(ctx)

File "/home/alex/miniconda3/envs/moseq2/lib/python3.6/site-packages/click/core.py", line 1137, in invoke

return _process_result(sub_ctx.command.invoke(sub_ctx))

File "/home/alex/miniconda3/envs/moseq2/lib/python3.6/site-packages/moseq2_pca/util.py", line 49, in invoke

return super(custom_command_class, self).invoke(ctx)

File "/home/alex/miniconda3/envs/moseq2/lib/python3.6/site-packages/click/core.py", line 956, in invoke

return ctx.invoke(self.callback, **ctx.params)

File "/home/alex/miniconda3/envs/moseq2/lib/python3.6/site-packages/click/core.py", line 555, in invoke

return callback(*args, **kwargs)

File "/home/alex/miniconda3/envs/moseq2/lib/python3.6/site-packages/moseq2_pca/cli.py", line 347, in apply_pca

mask_params=mask_params)

File "/home/alex/miniconda3/envs/moseq2/lib/python3.6/site-packages/moseq2_pca/pca/util.py", line 284, in apply_pca_dask

f.copy('/metadata/extraction', f_scores, name=metadata_name)

File "/home/alex/miniconda3/envs/moseq2/lib/python3.6/site-packages/h5py/_hl/group.py", line 493, in copy

copypl, base.dlcpl)

File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper

File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper

File "h5py/h5o.pyx", line 217, in h5py.h5o.copy

RuntimeError: Unable to copy object (destination object already exists)

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/dattalab/moseq2-pca/issues/51, or mute the thread https://github.com/notifications/unsubscribe-auth/AAEp18m-VUcF13glNSpB7DcYJToYAVBFks5vR5lngaJpZM4bWTs3 .

YudiXie commented 5 years ago

I think so because I had just reinstalled moseq-pca by running 

pip install git+https://github.com/dattalab/moseq2-pca.git

But the error still occurs.

jmarkow commented 5 years ago

This was hopefully addressed in #38 . I would try the following,

pip uninstall moseq2-pca
pip install --upgrade git+https://github.com/dattalab/moseq2-pca.git

To check which version you're running try,

pip freeze

You can paste the output of that command here so I can make sure you're using the latest version of everything.

YudiXie commented 5 years ago

I tried to reinstall but the problem was still there.

this is the results from pip freeze

(moseq2) alex@UchidaGPU:/media/alex/DataDrive1/MoSeqData/7day_preexposure_MoSeq$ pip freeze
autoregressive==0.1.1
bokeh==1.0.4
certifi==2018.11.29
chest==0.2.3
Click==7.0
cloudpickle==0.8.0
cycler==0.10.0
Cython==0.29.5
dask==1.1.2
dask-jobqueue==0.4.1
decorator==4.3.2
distributed==1.26.0
docrep==0.2.5
dtaidistance==1.1.4
future==0.17.1
h5py==2.9.0
hdf5storage==0.1.15
HeapDict==1.0.0
Jinja2==2.10
joblib==0.11
kiwisolver==1.0.1
locket==0.2.0
MarkupSafe==1.1.1
matplotlib==3.0.2
moseq2-batch==0.1.2
moseq2-extract==0.1.2
moseq2-model==0.1.2
moseq2-pca==0.1.2
moseq2-viz==0.1.2
msgpack==0.6.1
networkx==2.2
nose==1.3.7
numpy==1.14.5
opencv-python==4.0.0.21
packaging==19.0
pandas==0.24.1
partd==0.3.9
patsy==0.5.1
Pillow==5.4.1
psutil==5.5.1
pybasicbayes==0.2.4
pyhsmm==0.1.7
pyparsing==2.3.1
python-dateutil==2.8.0
pytz==2018.9
PyWavelets==1.0.2
PyYAML==3.13
ruamel.yaml==0.15.88
scikit-image==0.14.2
scikit-learn==0.19.0
scipy==1.2.1
seaborn==0.9.0
six==1.12.0
sklearn==0.0
sortedcontainers==2.1.0
statsmodels==0.9.0
tblib==1.3.2
toolz==0.9.0
tornado==5.1.1
tqdm==4.31.1
zict==0.1.3
jmarkow commented 5 years ago

Try deleting the ‘-pca’ directory and try everything from scratch. If that fails see if you can put your dataset in a Dropbox folder and share with me and I can have a quick look. Note that I’m out of the country for a week at a conference but will try to do what I can. On Fri, Mar 1, 2019 at 12:46 AM Yu Xie notifications@github.com wrote:

I tried to reinstall but the problem was still there.

this is the results from pip freeze

(moseq2) alex@UchidaGPU:/media/alex/DataDrive1/MoSeqData/7day_preexposure_MoSeq$ pip freeze autoregressive==0.1.1 bokeh==1.0.4 certifi==2018.11.29 chest==0.2.3 Click==7.0 cloudpickle==0.8.0 cycler==0.10.0 Cython==0.29.5 dask==1.1.2 dask-jobqueue==0.4.1 decorator==4.3.2 distributed==1.26.0 docrep==0.2.5 dtaidistance==1.1.4 future==0.17.1 h5py==2.9.0 hdf5storage==0.1.15 HeapDict==1.0.0 Jinja2==2.10 joblib==0.11 kiwisolver==1.0.1 locket==0.2.0 MarkupSafe==1.1.1 matplotlib==3.0.2 moseq2-batch==0.1.2 moseq2-extract==0.1.2 moseq2-model==0.1.2 moseq2-pca==0.1.2 moseq2-viz==0.1.2 msgpack==0.6.1 networkx==2.2 nose==1.3.7 numpy==1.14.5 opencv-python==4.0.0.21 packaging==19.0 pandas==0.24.1 partd==0.3.9 patsy==0.5.1 Pillow==5.4.1 psutil==5.5.1 pybasicbayes==0.2.4 pyhsmm==0.1.7 pyparsing==2.3.1 python-dateutil==2.8.0 pytz==2018.9 PyWavelets==1.0.2 PyYAML==3.13 ruamel.yaml==0.15.88 scikit-image==0.14.2 scikit-learn==0.19.0 scipy==1.2.1 seaborn==0.9.0 six==1.12.0 sklearn==0.0 sortedcontainers==2.1.0 statsmodels==0.9.0 tblib==1.3.2 toolz==0.9.0 tornado==5.1.1 tqdm==4.31.1 zict==0.1.3

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/dattalab/moseq2-pca/issues/51#issuecomment-468497971, or mute the thread https://github.com/notifications/unsubscribe-auth/AAEp19eFJC099oaDs3s9KC0LvHdbv-0Cks5vSHhlgaJpZM4bWTs3 .

jmarkow commented 5 years ago

Sorry ‘_pca’ On Fri, Mar 1, 2019 at 1:05 PM Jeffrey Markowitz markowitzmeister@gmail.com wrote:

Try deleting the ‘-pca’ directory and try everything from scratch. If that fails see if you can put your dataset in a Dropbox folder and share with me and I can have a quick look. Note that I’m out of the country for a week at a conference but will try to do what I can. On Fri, Mar 1, 2019 at 12:46 AM Yu Xie notifications@github.com wrote:

I tried to reinstall but the problem was still there.

this is the results from pip freeze

(moseq2) alex@UchidaGPU:/media/alex/DataDrive1/MoSeqData/7day_preexposure_MoSeq$ pip freeze autoregressive==0.1.1 bokeh==1.0.4 certifi==2018.11.29 chest==0.2.3 Click==7.0 cloudpickle==0.8.0 cycler==0.10.0 Cython==0.29.5 dask==1.1.2 dask-jobqueue==0.4.1 decorator==4.3.2 distributed==1.26.0 docrep==0.2.5 dtaidistance==1.1.4 future==0.17.1 h5py==2.9.0 hdf5storage==0.1.15 HeapDict==1.0.0 Jinja2==2.10 joblib==0.11 kiwisolver==1.0.1 locket==0.2.0 MarkupSafe==1.1.1 matplotlib==3.0.2 moseq2-batch==0.1.2 moseq2-extract==0.1.2 moseq2-model==0.1.2 moseq2-pca==0.1.2 moseq2-viz==0.1.2 msgpack==0.6.1 networkx==2.2 nose==1.3.7 numpy==1.14.5 opencv-python==4.0.0.21 packaging==19.0 pandas==0.24.1 partd==0.3.9 patsy==0.5.1 Pillow==5.4.1 psutil==5.5.1 pybasicbayes==0.2.4 pyhsmm==0.1.7 pyparsing==2.3.1 python-dateutil==2.8.0 pytz==2018.9 PyWavelets==1.0.2 PyYAML==3.13 ruamel.yaml==0.15.88 scikit-image==0.14.2 scikit-learn==0.19.0 scipy==1.2.1 seaborn==0.9.0 six==1.12.0 sklearn==0.0 sortedcontainers==2.1.0 statsmodels==0.9.0 tblib==1.3.2 toolz==0.9.0 tornado==5.1.1 tqdm==4.31.1 zict==0.1.3

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/dattalab/moseq2-pca/issues/51#issuecomment-468497971, or mute the thread https://github.com/notifications/unsubscribe-auth/AAEp19eFJC099oaDs3s9KC0LvHdbv-0Cks5vSHhlgaJpZM4bWTs3 .

YudiXie commented 5 years ago

Deleting _pca and starting from scratch didn't work. I will show you the data via email. Thank you!

jmarkow commented 5 years ago

OK bringing the conversation back here for everyone's benefit. Everything worked for me using version v0.1.2 (current master). It's possible that moseq2-pca is not updating currently via pip. Also, I don't recall if you tried this,

pip uninstall moseq2-pca
pip install --upgrade git+https://github.com/dattalab/moseq2-pca.git

You mentioned trying install without the --upgrade option, which may not have upgraded you to the latest.

A couple of things to try if that doesn't work for you:

  1. Create a new Conda environment (name it something like moseq2-test) and install everything from scratch per the latest instructions in the wiki.
  2. Try manually removing the moseq2-pca package in your current conda environment, then reinstall moseq2-pca
pip uninstall moseq2-pca
rm -rf ~/miniconda3/envs/moseq2/lib/python3.6/site-packages/moseq2_pca/
pip install --upgrade git+https://github.com/dattalab/moseq2-pca.git

Be careful with rm -rf since it will remove whatever directory you point it at, which is why you may want to just create a new environment to test things out first.

davidhbrann commented 5 years ago

As another option, if you want to make sure you really install everything fresh without running the rm -Rf command to remove other python packages that are not dependencies for moseq is to try pip install --upgrade --no-cache-dir --force-reinstall git+https://github.com/dattalab/moseq2-pca.git.

YudiXie commented 5 years ago

Tried all of the solutions and it still doesn't work.

  1. After running:

    pip uninstall moseq2-pca
pip install --upgrade git+https://github.com/dattalab/moseq2-pca.git
moseq2-pca apply-pca -c 6 -n 1

    Still got the same error message.

  2. After creating a new conda env and reinstall everything, still got the same error message.

  3. After running 

    pip install --upgrade --no-cache-dir --force-reinstall git+https://github.com/dattalab/moseq2-pca.git.
moseq2-pca apply-pca -c 6 -n 1

    Still got the same error message.

  4. After manually removing the moseq2-pca package and reinstalling again.

    pip uninstall moseq2-pca
rm -rf ~/miniconda3/envs/moseq2/lib/python3.6/site-packages/moseq2_pca/
pip install --upgrade git+https://github.com/dattalab/moseq2-pca.git
moseq2-pca apply-pca -c 6 -n 1

    Still got the same error message.

jmarkow commented 5 years ago

OK. Did you try deleting the _pca folder and training the PCs again from scratch? If that doesn't work I'll double check things on my end.

Either way I uploaded the _pca results into our shared Dropbox folder.

YudiXie commented 5 years ago

I will have a try and let you know. Thank you!

jmarkow commented 5 years ago

I was able to reproduce the issue, I think this bug was introduced by a recent pull request from another lab. I'm fixing it now.

jmarkow commented 5 years ago

I found the problem, will be updating master in the next 10 minutes with a fix.

jmarkow commented 5 years ago

Okay. Terribly sorry I didn't catch this before, this was my fault for merging in a change before it was fully vetted. Try updating one more time and it should work now, just tested myself.

pip uninstall moseq2-pca
pip install --upgrade git+https://github.com/dattalab/moseq2-pca.git
YudiXie commented 5 years ago

Problem solved! Thank you so much!