search

VoVAllen / tf-dlpack

DLPack for Tensorflow
Apache License 2.0
36 stars 4 forks source link

kindly remove tensorflow requirement in pip and also provide wheels for other OS #11

Open Abhijit-2592 opened 4 years ago

Abhijit-2592 commented 4 years ago

Hey thanks for the awesome project. I see that the pip install also forcibly installs tensorflow and causes problems with already installed versions of tensorflow. It will be great if you can remove Tensorflow Dependency from the pip install. Also it will be great to add wheels for other OS too. We are looking to add this project as a dependency to Thinc. As of now it causes the above problems while we install.

honnibal commented 4 years ago

Just to clarify quickly: the issue is that we might have tensorflow-gpu installed, rather than the CPU tensorflow. Unfortunately TF makes life really hard here :(.

VoVAllen commented 4 years ago

Sure. I'll make it optional, as I also found someone using tf-nightly version.

Abhijit-2592 commented 4 years ago

@VoVAllen Thanks. Is there a reason why windows isn't supported? And it will be great if you could also release wheels for mac os

honnibal commented 4 years ago

@Abhijit-2592 Just to step in with some perspective from maintaining a different project :).

MacOS wheels are likely really hard, for the same reason that they're really pretty useless: it's very unusual to have an NVIDIA GPU attached to a MacOS machine and be doing machine learning on it. Because nobody does this, the setup for it is kind of unclear. And it's not like you can easily rent a MacOS VM...So to develop and debug this, you'd probably have to sit down with a Mac laptop, attach a GPU with an external enclosure, and build the wheels manually. And then you'd have to do the same for every release...

Considerations around Windows are similar: scripting anything on Windows really sucks, and renting and setting up cloud VMs to debug it really sucks too. Azure Pipelines makes it a bit easier, but the process is still pretty hard. It's reasonable for a developer to say, I don't use Windows, I probably never will, nobody in my team or even that I really know uses Windows, and to do this I'd have to do a bunch of work...Why is this my problem?

The fact is that the maintenance burden of building cross-platform wheels for this project is potentially going to be really bad. The code itself might not change at all, but new versions will need to be built as new Tensorflows are released. And Tensorflow is released pretty often. A release process that involves logging into different machines and manually building wheels won't really be viable. That'll be really hard to maintain. Consider that even with just Windows and Linux support, you're building 6 wheels per release (for Python 3.6, 3.7 and 3.8).

honnibal commented 4 years ago

Okay so, after all that about the problems, here's what I think we should do for spaCy/Thinc.

I think the best solution for now would be for us to develop a recipe for conda-forge. It's pretty easy to work with, and it comes with all the cross-platform automation built in. It also lets Explosion take care of routine maintenance if we need it.

To me if the project supports conda and pip on Linux, that's totally reasonable. Most Windows users use conda anyway...

Again, thanks @VoVAllen for your work on this. It's disappointing that the TF team has left this important inter-operability to the community...It seems like it should be very easy for them to do, while it's much harder to do as a separate package.

Abhijit-2592 commented 4 years ago

@honnibal thanks for throwing light on this. And @VoVAllen thanks for this awesome work!

VoVAllen commented 4 years ago

Actually we are working with Tensorflow team to put into Tensorflow framework, which you can find at https://github.com/tensorflow/community/pull/180. For the wheel, it's just because we don't have mac machine and compiling C++ is really complex on the windows. As this is an experimental version, we just released few binary versions. We hope we could bring this to the official tensorflow soon. Thanks for your suggestions.