search

ratt-ru / shadeMS

Rapid Measurement Set plotting with dask-ms and datashader
21 stars 6 forks source link

Pandas error while trying to run shadems on ilifu #111

Open KshitijT opened 1 year ago

KshitijT commented 1 year ago

I am getting this error while trying to run shadems installed in a virtualenvironment in ilifu:

Traceback (most recent call last): File "/idia/users/kshitij/shadems_venv3/bin/shadems", line 6, in <module> from shade_ms import main File "/idia/users/kshitij/shadems_venv3/lib/python3.8/site-packages/shade_ms/main.py", line 25, in <module> from . import data_plots, data_mappers File "/idia/users/kshitij/shadems_venv3/lib/python3.8/site-packages/shade_ms/data_plots.py", line 10, in <module> import dask.dataframe as dask_df File "/idia/users/kshitij/shadems_venv3/lib/python3.8/site-packages/dask/dataframe/__init__.py", line 4, in <module> from dask.dataframe import backends, dispatch, rolling File "/idia/users/kshitij/shadems_venv3/lib/python3.8/site-packages/dask/dataframe/backends.py", line 22, in <module> from dask.dataframe.core import DataFrame, Index, Scalar, Series, _Frame File "/idia/users/kshitij/shadems_venv3/lib/python3.8/site-packages/dask/dataframe/core.py", line 35, in <module> from dask.dataframe import methods File "/idia/users/kshitij/shadems_venv3/lib/python3.8/site-packages/dask/dataframe/methods.py", line 22, in <module> from dask.dataframe.utils import is_dataframe_like, is_index_like, is_series_like File "/idia/users/kshitij/shadems_venv3/lib/python3.8/site-packages/dask/dataframe/utils.py", line 19, in <module> from dask.dataframe import ( # noqa: F401 register pandas extension types File "/idia/users/kshitij/shadems_venv3/lib/python3.8/site-packages/dask/dataframe/_dtypes.py", line 4, in <module> from dask.dataframe.extensions import make_array_nonempty, make_scalar File "/idia/users/kshitij/shadems_venv3/lib/python3.8/site-packages/dask/dataframe/extensions.py", line 6, in <module> from dask.dataframe.accessor import ( File "/idia/users/kshitij/shadems_venv3/lib/python3.8/site-packages/dask/dataframe/accessor.py", line 190, in <module> class StringAccessor(Accessor): File "/idia/users/kshitij/shadems_venv3/lib/python3.8/site-packages/dask/dataframe/accessor.py", line 276, in StringAccessor pd.core.strings.StringMethods, AttributeError: module 'pandas.core.strings' has no attribute 'StringMethods'

Probably something to do with nocoffeebrain, but I can't figure out where the error is coming from; the Shadems installed via CARACal works fine when running the pipeline. In the above instance, shadems was installed simply with pip install shadems.

o-smirnov commented 1 year ago

Most likely a newer version of pandas has broken something in the API. Could you try pip installing a few older versions of pandas into the same venv to see if that resolves the problem?

KshitijT commented 1 year ago

Pandas are being evil with version 0.5.0 too....

hrkloeck commented 1 year ago

Hi both I'm running into the same problem, so what kind of steps are needed to get it properly installed.

KshitijT commented 1 year ago

Hi both I'm running into the same problem, so what kind of steps are needed to get it properly installed.

I still haven't been able to resolve this, but the CARACal shadems works fine, so if you have CARACal installed in a virtualenv, perhaps you could use that version instead?

hrkloeck commented 1 year ago

Thanks for the reply. I’m trying to build a singularity container, maybe you know of the version id’s of the used packages I could try build it.

On 28. Apr 2023, at 14:46, Kshitij Thorat @.***> wrote:

Hi both I'm running into the same problem, so what kind of steps are needed to get it properly installed.

I still haven't been able to resolve this, but the CARACal shadems works fine, so if you have CARACal installed in a virtualenv, perhaps you could use that version instead?

— Reply to this email directly, view it on GitHub https://github.com/ratt-ru/shadeMS/issues/111#issuecomment-1527511589, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALCUM5LY76BH5BKTSE4PKUDXDO3Z5ANCNFSM6AAAAAAW2OEGOA. You are receiving this because you commented.

IanHeywood commented 1 year ago

Hi Hans, shadems definitely works in this container:

https://github.com/IanHeywood/oxkat/blob/dev/README.md#containers

You might be able to use that for your purposes or at least start the container in a shell and deduce which dependency versions you need for your own.

On Fri, 28 Apr 2023, 14:34 hrkloeck, @.***> wrote:

Thanks for the reply. I’m trying to build a singularity container, maybe you know of the version id’s of the used packages I could try build it.

Hans-Rainer Klöckner

Max-Planck-Institut für Radioastronomie Auf dem Hügel 69 53121 Bonn Germany +49 (0)228 525-338 @.***

On 28. Apr 2023, at 14:46, Kshitij Thorat @.***> wrote:

Hi both I'm running into the same problem, so what kind of steps are needed to get it properly installed.

I still haven't been able to resolve this, but the CARACal shadems works fine, so if you have CARACal installed in a virtualenv, perhaps you could use that version instead?

— Reply to this email directly, view it on GitHub < https://github.com/ratt-ru/shadeMS/issues/111#issuecomment-1527511589>, or unsubscribe < https://github.com/notifications/unsubscribe-auth/ALCUM5LY76BH5BKTSE4PKUDXDO3Z5ANCNFSM6AAAAAAW2OEGOA . You are receiving this because you commented.

— Reply to this email directly, view it on GitHub https://github.com/ratt-ru/shadeMS/issues/111#issuecomment-1527573412, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABR3FJ5GFO6D32YRSHJX4VDXDPBQBANCNFSM6AAAAAAW2OEGOA . You are receiving this because you are subscribed to this thread.Message ID: @.***>

hrkloeck commented 1 year ago

perfect I will try, thanks a lot.

On 28. Apr 2023, at 17:03, IanHeywood @.***> wrote:

Hi Hans, shadems definitely works in this container:

https://github.com/IanHeywood/oxkat/blob/dev/README.md#containers

You might be able to use that for your purposes or at least start the container in a shell and deduce which dependency versions you need for your own.

On Fri, 28 Apr 2023, 14:34 hrkloeck, @.***> wrote:

Thanks for the reply. I’m trying to build a singularity container, maybe you know of the version id’s of the used packages I could try build it.

On 28. Apr 2023, at 14:46, Kshitij Thorat @.***> wrote:

Hi both I'm running into the same problem, so what kind of steps are needed to get it properly installed.

I still haven't been able to resolve this, but the CARACal shadems works fine, so if you have CARACal installed in a virtualenv, perhaps you could use that version instead?

— Reply to this email directly, view it on GitHub < https://github.com/ratt-ru/shadeMS/issues/111#issuecomment-1527511589>, or unsubscribe < https://github.com/notifications/unsubscribe-auth/ALCUM5LY76BH5BKTSE4PKUDXDO3Z5ANCNFSM6AAAAAAW2OEGOA . You are receiving this because you commented.

— Reply to this email directly, view it on GitHub https://github.com/ratt-ru/shadeMS/issues/111#issuecomment-1527573412, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABR3FJ5GFO6D32YRSHJX4VDXDPBQBANCNFSM6AAAAAAW2OEGOA . You are receiving this because you are subscribed to this thread.Message ID: @.***>

— Reply to this email directly, view it on GitHub https://github.com/ratt-ru/shadeMS/issues/111#issuecomment-1527704682, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALCUM5NALOAOAMXF6WTH2NDXDPL5NANCNFSM6AAAAAAW2OEGOA. You are receiving this because you commented.

Athanaseus commented 1 year ago

Hi @hrkloeck, can you please try: pip install shadems -U Made some updates to the dependency list.

hrkloeck commented 1 year ago

Hi Athanaseus,

I have tried to install it on a singularity ubuntu 20.04

and shadems -h produced the following

pkg_resources.ContextualVersionConflict: (xarray 2023.1.0 (/usr/local/lib/python3.8/dist-packages), Requirement.parse('xarray<2023.0.0,>=2022.6.0; extra == "xarray" or extra == "complete"'), {'dask-ms'})

Athanaseus commented 1 year ago

Does it help to create a new virtual environment? Also, did you get any warning while installing?

IanHeywood commented 1 year ago

Also @hrkloeck are you installing anything else that has a dask-ms dependency in the same container recipe? I've run into this issue often where two things in the same container have different dependency versions, and the installation of one breaks the other.

hrkloeck commented 1 year ago

thanks for checking I completely cleaned it up and just have that line in my installation

pip3 install shadems -U

On 2. May 2023, at 16:04, IanHeywood @.***> wrote:

Also @hrkloeck https://github.com/hrkloeck are you installing anything else that has a dask-ms dependency in the same container recipe? I've run into this issue often where two things in the same container have different dependency versions, and the installation of one breaks the other.

— Reply to this email directly, view it on GitHub https://github.com/ratt-ru/shadeMS/issues/111#issuecomment-1531541835, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALCUM5IQNVLAF7O5A7ASGBTXEEH75ANCNFSM6AAAAAAW2OEGOA. You are receiving this because you were mentioned.

hrkloeck commented 1 year ago

Dear all, thanks for the feedback. I could sort it out.

These are the lines I need to place as early as possible within the singularity file

pip3 install xarray==2022.6.0 pip3 install dask-ms[xarray] pip3 install shadems -U

Cheers, Hans