search

neptune-ai / neptune-client

📘 The experiment tracker for foundation model training
https://neptune.ai
Apache License 2.0
584 stars 63 forks source link

BUG: Unable to install neptune-client in kaggle kernel #1268

Closed gfx73 closed 1 year ago

gfx73 commented 1 year ago

neptune-client problem I tried to install neptune-clinet couple of monthes ago and is was just fine with "!pip install neptune-client".

The error I have: ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. aiobotocore 2.4.2 requires botocore<1.27.60,>=1.27.59, but you have botocore 1.29.79 which is incompatible.

What I've done: 1) I'm doing the installation right after factory reset in "run" tab.

2) I tried all versions of neptune-client from 0.16.11 to 1.0.0 (which is just neptune e.g. "pip install neptune")

3) As I understand neptun-client requires aiobotocore-2.4.2, aiobotocore 2.4.2 requires botocore-1.27.59. So I tried to search for some boto3 version which is compatilble. The newest compatible version I found is boto3==1.24.59. Kaggle kernel has preinstalled versions of botocore and boto3. So I tried to install neptune-client in the following way:

!pip uninstall botocore boto3 -y
!pip install aiobotocore botocore boto3==1.24.59 neptune-client

The installation itself was ok, without any erros. However at this point I got an error when trying to initialize NeptuneLogger: ModuleNotFoundError: FileNotFoundError: [Errno 2] No such file or directory: '/opt/conda/lib/python3.7/site-packages/boto3-1.26.72.dist-info/METADATA'. HINT: Try running pip install -U 'neptune-client'. So it seems like neptune-client needs boto3-1.26.72, which is incompatible with aiobotocore-2.4.2). Maybe I'm wrong in something. Just tried to figure out what is the source of problem.

4) It looks like this problem arrived with some kaggle environment update. I have "pin original environment" option in notebooks settings. In one notebook I have version dated by 2023-02-01. In this notebook I can just install neptune-client and everything is fine. However when I switch to "Always use the latest environment" I have the described problem. My new notebook's environment is dated by 2023-02-17. This notebook also have the described problem, and I can't switch to older environment version.

To reproduce just try !pip install neptune-client in a kaggle notebook with new environment version.

I am sorry if the issue is related not to neptune-client, but to kaggle. Anyway, I would really appreciate any help)

Blaizzy commented 1 year ago

Hey @gfx73

Prince here,

Thanks for reaching out!

Could you try again?

I tried to reproduce your issue but it worked without any issues.

gfx73 commented 1 year ago

Hi @Blaizzy

Thanks for reply.

I created a new notebook and tried to install. The installation was successful indeed. However, once I switched environment option I am not able to install neither with “pin original environment” nor with “Always use the latest environment”. I am doing factory reset before trying.

gfx73 commented 1 year ago

Sorry. Closed issue because of miss click.

Blaizzy commented 1 year ago

I created a new notebook and tried to install. The installation was successful indeed. However, once I switched environment option I am not able to install neither with “pin original environment” nor with “Always use the latest environment”. I am doing factory reset before trying

Are you able to install other packages besides Neptune?

gfx73 commented 1 year ago

Yes. I installed faiss for example

Blaizzy commented 1 year ago

However, once I switched environment option I am not able to install neither with “pin original environment” nor with “Always use the latest environment”. I am doing factory reset before trying

Could you elaborate on this? What do you mean by changing environments? How can I reproduce this state?

gfx73 commented 1 year ago

img Environment option on the right side.

I'm sorry again if the issue is on kaggle's side.

Blaizzy commented 1 year ago

Unfortunately, I am unable to reproduce your issue. I changed my environment and installed neptune again, and it works fine. Screenshot 2023-03-01 at 10 55 53 AM

Blaizzy commented 1 year ago

After a deeper look, I see the error now.

But it doesn't affect your use of neptune.

I will ask the devs to investigate this issue for you and see if there is anything we can do here. Because, by the looks of Kaggle has the latest botocore installed, which results in the requirement being satisfied when installing neptune.

gfx73 commented 1 year ago

img

I can't initialize NeptuneLogger.

Blaizzy commented 1 year ago

What version of neptune are you running?

gfx73 commented 1 year ago

The default which I got from pip. neptune.__version__ returns 1.0.2

Blaizzy commented 1 year ago

We are currently working on updating the Lightning integration to support v1.0.0 changes and potentially will be available on the next release of Lightning.

You can follow it here: https://github.com/Lightning-AI/lightning/pull/16761

Meanwhile, the devs work on the update and a fix for the issue you reported, you can temporarily downgrade to neptune-client to v0.16.17

pip install neptune-client==0.16.17

This will allow you to continue working :)

Blaizzy commented 1 year ago

Let me know if this helps you

gfx73 commented 1 year ago

Of course. Just a second, I'm trying

gfx73 commented 1 year ago

Well, I have the same issue img

Blaizzy commented 1 year ago

Can you downgrade botocore?

gfx73 commented 1 year ago

So should I install this way: pip install botocore==1.27.59 neptune-client==0.16.17?

Blaizzy commented 1 year ago

Please give it a try

gfx73 commented 1 year ago

Ok, I tried. Error remains. I have an interesting thing to add... img

Blaizzy commented 1 year ago

Try to restart the environment after you make the installation.

gfx73 commented 1 year ago

I had this error with botocore3==1.27.59: "ModuleNotFoundError: FileNotFoundError: [Errno 2] No such file or directory: '/opt/conda/lib/python3.7/site-packages/boto3-1.26.77.dist-info/METADATA'. HINT: Try running `pip install -U 'neptune-client'"

So I tried to install boto3==1.26.77. But it's not compatible with botocore==1.27.59.

I actually tried to resolve dependencies before creating the issue.

Blaizzy commented 1 year ago

There is a potential workaround. Could you try the following:

  1. Uninstall aibotocore, botocore, and s3fs.
  2. Afterward, try to install neptune and s3fs.
gfx73 commented 1 year ago

It didn't work

Blaizzy commented 1 year ago

What output are you getting?

gfx73 commented 1 year ago

img

Blaizzy commented 1 year ago

Thanks! I have sent it to the devs so they can investigate this issue for you.

gfx73 commented 1 year ago

Thanks for help!

Blaizzy commented 1 year ago

Could you try one more sequence?

  1. Install neptune
  2. Uninstall aiobotocore, botocore, and s3fs
  3. Reinstall s3fs
gfx73 commented 1 year ago

ok

gfx73 commented 1 year ago

When installing neptune-client: "ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. boto3 1.26.77 requires botocore<1.30.0,>=1.29.77, but you have botocore 1.27.59 which is incompatible."

When installing s3fs: "ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. aiobotocore 2.4.2 requires botocore<1.27.60,>=1.27.59, but you have botocore 1.29.81 which is incompatible."

And this one when initializing logger: "ModuleNotFoundError: ContextualVersionConflict: (botocore 1.27.59 (/opt/conda/lib/python3.7/site-packages), Requirement.parse('botocore<1.30.0,>=1.29.77'), {'boto3'}). HINT: Try running pip install -U 'neptune-client'"

Blaizzy commented 1 year ago

Did you execute step 2?

gfx73 commented 1 year ago

Yep

Blaizzy commented 1 year ago

Hi, I just heard from the devs that the process should be !pip install boto3==1.24.40 Then: !pip install neptune-client

Let me know if this helps

Blaizzy commented 1 year ago

Hey there,

Just checking to see if you still need help with this :)

gfx73 commented 1 year ago

Hi.

I think the only thing I can do is to wait for update. So I currently switched to another logger. However, I like Neptune more and would be glad to continue using it)

Blaizzy commented 1 year ago

Hey @gfx73

I talked to the developers, and it turns out that the issue lies with the Kaggle kernels rather than the Neptune client.

You'll have to wait for them to resolve the problem, or you can try to file an issue on their repository. In the meantime, the best option we have is this one: https://github.com/neptune-ai/neptune-client/issues/1268#issuecomment-1451620163

gfx73 commented 1 year ago

Thank you. Should I close the issue?

gfx73 commented 1 year ago

This works https://github.com/neptune-ai/neptune-client/issues/1268#issuecomment-1451620163

Blaizzy commented 1 year ago

Perfect, if anything else pops up, do let me know, and I'll be happy to help.

Have a good day! ☀️

gfx73 commented 1 year ago

Thank you for help.

Have a good day)