search

bigmlcom / sensenet

0 stars 8 forks source link

Tensorflow and tensorflow-gpu dependencies #18

Closed unmonoqueteclea closed 2 years ago

unmonoqueteclea commented 3 years ago

Although tensorflow and tensorflow-gpu are the same package, pip is not able to know. We are requiring tensorflow so, if user installed tensorflow-gpu (for instance GPU Tensorflow Dockerfiles come with tensorflow-gpu installed), pip will try to install tensorflow package too.

This doesn't seem to have an easy solution, but at least we could consider some of the solutions proposed here: https://github.com/tensorflow/tensorflow/issues/7166

charleslparker commented 3 years ago

Although tensorflow and tensorflow-gpu are the same package, pip is not able to know.

Wait, do you mean that if you run, for example, pip freeze | grep tensorflow, you see the package tensorflow-gpu but not tensorflow? Or do you see tensorflow and pip just does not understand it is tensorflow? Or do you see neither thing, but tensorflow is installed just the same?

Unfortunately, tensorflow/tensorflow#7166 may not be able to solve this problem, because tensorflow is a build tool here and the version is also given in pyproject.toml . . . but maybe this version doesn't matter because it's only installed for builds and doesn't stay on the system? I'm maybe a bit confused about how pyproject.toml works exactly.

Anyway, does this actually break things for you, or is it just doing unnecessary installations? I can probably put some hacks in setup.py as long as the download that may happen in pyproject.toml is not a problem. It would be much harder to put this logic in Pyramid because @petersen-poul parses the dependencies there by hand to build the bigml server images, so hopefully this is not needed in pyramid.

unmonoqueteclea commented 3 years ago

Wait, do you mean that if you run, for example, pip freeze | grep tensorflow, you see the package tensorflow-gpu but not tensorflow? Or do you see tensorflow and pip just does not understand it is tensorflow? Or do you see neither thing, but tensorflow is installed just the same?

By default I have pre-installed tensorflow-gpu in my image (I see it when I do pip freeze | grep tensorflow) but pip sees the requirement tensorflow and doesn't know that it is already installed so it installs also tensorflow instead using tensorflow-gpu

Unfortunately, tensorflow/tensorflow#7166 may not be able to solve this problem, because tensorflow is a build tool here and the version is also given in pyproject.toml . . . but maybe this version doesn't matter because it's only installed for builds and doesn't stay on the system? I'm maybe a bit confused about how pyproject.toml works exactly.

Anyway, does this actually break things for you, or is it just doing unnecessary installations? I can probably put some hacks in setup.py as long as the download that may happen in pyproject.toml is not a problem. It would be much harder to put this logic in Pyramid because @petersen-poul parses the dependencies there by hand to build the bigml server images, so hopefully this is not needed in pyramid.

Now I think it's only doing unnecessary installations but I am having problems with other dependencies that I am trying to fix one by one and I wasn't able yet to check if it is using the GPU. I am installing Pyramid with pip, should I install it in a different way to use pyproject.toml?

charleslparker commented 3 years ago

By default I have pre-installed tensorflow-gpu in my image (I see it when I do pip freeze | grep tensorflow)

So when they built the image, Google used a non-public package with a different name. Nice.

I am installing Pyramid with pip, should I install it in a different way to use pyproject.toml ?

No, pip uses pyproject.toml. You'll see the step installing build dependencies ... when you run pip install, and here I think it grabs the version of tensorflow in pyproject.toml and uses it to build the package but does not actually install it. That is, I think the build environment for the package is kept separate from the main environment. This should be fine for you, as the version it downloads/builds against should at least be compatible with your docker version, and does not (I think) overwrite it.

I THINK. MAYBE.

I'll put these hacks in setup to not install if the right TF version is already there, according to tensorflow.__version__.

sarusso commented 2 years ago

Note that now we also have tensorflow-macos and tensorflow-aarch64 to add more fun to the game.

charleslparker commented 2 years ago

This should be solved by the wheels being built and uploaded in https://github.com/bigmlcom/sensenet/pull/33, so I'm declaring victory here.