search

open-ce / transformers-feedstock

Apache License 2.0
0 stars 10 forks source link

[Question]: should transformer models be installed with conda or downloaded externally? #12

Closed adammoody closed 3 years ago

adammoody commented 3 years ago

I noticed the following message in my transformers build output:

Neither PyTorch nor TensorFlow >= 2.0 have been found.  Models won't be available and only tokenizers, configuration and file/data utilities can be used.

And I see that neither pytorch nor tensorflow are listed as package dependencies in the transformers recipie/meta.yaml file.

Is the goal here to just install a subset of things from transformers and have people fetch the transformer models through other means like github?

Thanks!

jayfurmanek commented 3 years ago

Hi @adammoody. Thanks for all of the feedback! The goal here was to allow transformers to build without having to build TensorFlow first (which is very time consuming). The message you are seeing there is from the TEST section of the conda build where it does a python import to ensure the resultant package is sound. In normal practice if you see this, and then install TensorFlow and/or Pytorch the message goes away. TensorFlow is not needed at build time to enable TensorFlow integration (this stuff is all just python).

Further, the conda build TEST section doesn't test extensive integration function and you can run some transformers things without TensorFlow (ex. with scikit) so we left it out of the run requires section.

We could add it in there to make that particular message go away, but I think it would be better to add TensorFlow and PyTorch tests to https://github.com/open-ce/transformers-feedstock/blob/master/tests/open-ce-tests.yaml.

adammoody commented 3 years ago

Ah, I see. When I saw that in the output, I was worried that I might have something wrong in my transformers build. Thanks for the explanation.

Thanks to you and the team for putting all of this together, and thanks for your help. It's been a bit bumpy due to differences on our system here, but at this point, we've got sufficient fixes and work arounds that all of the packages build.