search

matrix-profile-foundation / matrixprofile

A Python 3 library making time series data mining tasks, utilizing matrix profile algorithms, accessible to everyone.
https://matrixprofile.org
Apache License 2.0
360 stars 62 forks source link

Deploy matrixprofile library through conda #70

Closed vanbenschoten closed 3 years ago

vanbenschoten commented 3 years ago

Is your feature request related to a problem? Please describe. Currently, conda users have no easy way of downloading the matrixprofile library. This is a problem both from an accessibility standpoint, and for various initiatives on the MPF roadmap.

Describe the solution you'd like matrixprofile can be downloaded through conda-install matrixprofile

Describe alternatives you've considered N/A

Additional context N/A

I'll be taking this task in tandem with @frankiecancino , unless other members of the community would like to help :)

nils-braun commented 3 years ago

Thanks for bringing this up, @vanbenschoten! I just want to mention that I am happy to support here (as I have already created two conda-forge feedstock repositories, for dask-sql or tsfresh), if you need any help

vanbenschoten commented 3 years ago

@nils-braun Will do! Appreciate the offer.

vanbenschoten commented 3 years ago

I'll be working on the conda-forge deployment here

vanbenschoten commented 3 years ago

Ok, everything should be set in the 'matrixprofile' branch for upload - the only thing remaining is the SHA256 key. @tylerwmarrs do you know how we can get this from PyPi? I thought it would be under the general project description but I can't seem to locate it (also, feel free to look through the meta.yaml file as I added you as a conda-forge maintainer :)

Once the key is added, we'll be ready to make a PR to conda-forge!

tylerwmarrs commented 3 years ago

@vanbenschoten here is where you can get them. Notice that there is a key for EVERY package.

image

I found the instructions on this page:

https://conda-forge.org/docs/maintainer/adding_pkgs.html#meta-yaml

image

vanbenschoten commented 3 years ago

@nils-braun do you have any experience working with the build.sh or bld.bat files? Because matrixprofile leverages Cython and OpenMP, there are essential dependencies that need to be installed on the build machine.

Alternatively, I'm trying to find documentation on whether it's possible to upload multiple packages (as referenced by Tyler above). That might be easier, but the one relevant link I found on conda was dead :(

nils-braun commented 3 years ago

So I have never used build.sh, but I am not sure if you actually need it. Just to be sure we are all on the same page (I guess you have already read through the documentation, but let me quickly repeat):

So, in short: add your dependencies to the requirements.build section in the meta.yaml and I think you will not need any build scripts separately.

Hope that made it a bit clearer, if not: I am happy to answer any additional questions It might also make sense to have a look into other "more complicated" projects, e.g. pandas which also needs to do some C compilations

vanbenschoten commented 3 years ago

Thanks @nils-braun! Super helpful. I had been looking at a couple of other complex repos but hadn't caught the importance of the build.requirements section of meta.yaml vs. build.sh.

I'm going to take another pass at this later today.

On Fri, Jan 8, 2021, 10:37 AM Nils Braun notifications@github.com wrote:

So I have never used build.sh, but I am not sure if you actually need it. Just to be sure we are all on the same page (I guess you have already read through the documentation, but let me quickly repeat):

  • You out into the meta.yaml the source repository from pipy (the tar.gz.). On a release, you upload a new pipy tar.gz source repo (together with the wheel that you already upload I guess). This is the URl that needs to go into the meta.yaml. As I have seen, you have already done so
  • You have also solved the checksum issue. Cool! Later, you will not need to do this again as conda-forge will actually notice if you have uploaded a new package to pipy and create a PR to fix it automatically (quite smart I think). You can also download from github releases if this is easier for you.
  • This means, the source is the same for all platforms, no matter if you have included any non-python files. You should not include noarch: python, because otherwise also the output will be the same on all platforms (but you do not want that)
  • You add you build requirements in the meta.yaml (as here https://github.com/conda-forge/staged-recipes/blob/master/recipes/example/meta.yaml#L38). There is some information https://conda-forge.org/docs/maintainer/knowledge_base.html#openmp on OPenMP.
  • Now concerning the build file: do you need to do more than just "pip install" (or python setup.py install)?. If not, you can keep the example line https://github.com/conda-forge/staged-recipes/blob/master/recipes/example/meta.yaml#L35 .

So, in short: add your dependencies to the requirements.build section in the meta.yaml and I think you will not need any build scripts separately.

Hope that made it a bit clearer, if not: I am happy to answer any additional questions It might also make sense to have a look into other "more complicated" projects, e.g. pandas https://github.com/conda-forge/pandas-feedstock/blob/master/recipe/meta.yaml which also needs to do some C compilations

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/matrix-profile-foundation/matrixprofile/issues/70#issuecomment-756858786, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB53ISBFQC7MZ3LN6XTJ5PDSY4YFBANCNFSM4VWYEXIA .

tylerwmarrs commented 3 years ago

@vanbenschoten I made some changes to the setup.py that should make the conda build easier. I also updated the recipes to link numpy and other cython related things as needed. I'll test later on to see if this resolves the issues. This required a version bump to get the source distribution on pypi to 1.1.8.

Basically, the setup.py included relative imports and file manipulation, however, the way conda references files was making it blow up.

tylerwmarrs commented 3 years ago

There is still an issue with importing the "version.py" file into "setup.py" during the conda build. I tested this locally in a similar fashion to the calls that conda makes and it works, but when deploying to conda, it fails.

The linux dependencies are met for libomp, but macosx and windows need something else.

tylerwmarrs commented 3 years ago

I resolved the conda building issues. I noticed that they do not build for python 3.9. I wonder when they will support this. Right now only 3.6, 3.7, and 3.8 are building.

image

tylerwmarrs commented 3 years ago

@nils-braun and @vanbenschoten everything for conda is complete. Need conda to merge and we can close this issue.

https://github.com/conda-forge/staged-recipes/pull/13626

vanbenschoten commented 3 years ago

@nils-braun do we need to tag someone on the conda-forge PR request, or will they automatically merge now that all tests pass?

nils-braun commented 3 years ago

Nice to see this progressing! Typically there is no need to tag someone, they are rather quick. See https://conda-forge.org/docs/maintainer/adding_pkgs.html#feedback-and-revision if they really do not react (but maybe give them some time)

tylerwmarrs commented 3 years ago

My conda deployment was merged. Closing.