Produce a source distribution file to pypi (tar.gz format)

[ChrisCarini]

Is your feature request related to a problem? Please describe. In some corporate environments, it's common to import external libraries into an internal repository for internal dependency resolution. In order to import matrixprofile for use, a tar.gz file is needed similar to the python libraries requests or pytest.

Describe the solution you'd like A tar.gz file uploaded to pypi as part of the release. See requests or pytest as examples. Links below.

Describe alternatives you've considered None.

Additional context

• https://pypi.org/project/requests/#files
• https://pypi.org/project/pytest/#files

[tylerwmarrs]

Thank you for raising this issue. The packages distributed on PyPi are currently binary wheels to avoid complications with dependencies during installation. A "tar.gz" deployment is distributing the source code. It is my understanding that some companies prefer to have the "tar.gz" file purely to execute some vulnerability scanning on the code. The "tar.gz" is the older way of distributing a library using "sdist" instead of the newer "wheel" format. See https://www.python.org/dev/peps/pep-0427/#id5

The two packages that you linked can easily distribute source code and installation is simplistic. This is because they are using pure Python while our library also uses Cython for speed reasons. Numpy is in a similar situation as us where the installation may be cumbersome for end-users when distributing source code. However, they do provide their source code as a ".zip".

Do you know if a ".zip" is suitable for your employer or is the older sdist (.tar.gz) source required?

For your reference, our documentation shows how to install from source on various operating systems: https://matrixprofile.docs.matrixprofile.org/install.html

[tylerwmarrs]

@ChrisCarini are you able to provide more insight?

[ChrisCarini]

Hey @tylerwmarrs - apologies, I missed the notification for your first detailed (thank you, by the way! :) ) response.

I'm confirming on the .zip vs .tar.gz - I believe a .zip should be fine, but am waiting on confirmation.

I do know my employer is getting support soon for consuming the wheel format, however, it will require a longer process to bring in, as it will require our company to manually sign off from a legal and security perspective (whereas, this process is simplified / more automated for sdists).

For the time being, I can manually pull this in for experimenting locally - let me get back to you regarding the .zip vs .tar.gz though!

And, thank you again for the detail above, python packaging is a bit out of my wheelhouse (see what I did there?! ha ha), so this helps me get more familiar with it!

[tylerwmarrs]

@ChrisCarini did you determine if the zip file works for your employer? Either way, I think we should probably add the source distribution as described above.

[ChrisCarini]

Hi @tylerwmarrs - confirmed with my employer, so long as it is a sdist, we should be able to bring it in for use.

We are working on a way to bring in wheels, although that is not as automated, yet (it requires manual approvals from our security and legal teams).

[tylerwmarrs]

@ChrisCarini thank you for figuring this out. I'll update the title to reflect the need for a "zip" file.

[ChrisCarini]

Thank you! And happy holidays!

[tylerwmarrs]

@ChrisCarini happy holidays!

@lmmentel this is another easy one if you have time. :)

[lmmentel]

Sure, it is actually possible to have both ".tar.gz" and ".zip" according to the docs. I can add both options to travis build.

[tylerwmarrs]

I started working on this myself as I found time to do it. :) It probably makes sense for me to tackle it since I have ownership of the TravisCI etc.

[tylerwmarrs]

@lmmentel you actually can only upload one archive type to PyPi. See screenshot:

[lmmentel]

I didn't know PyPI had this limitation. Do you want me to update the PR or you want to take it from here?

[tylerwmarrs]

I got it. Thanks for your help today! I only have to work a half day today and was going to tackle some of these "easy" issues. It is always nice to have help!

[lmmentel]

Glad to be useful.

I guess I thought that PyPI is more like github releases allowing multiple source formats.

BTW @ChrisCarini any reason you couldn't use source distributions released on github?

[ChrisCarini]

Wow, that was a fast turn around! Thank you!!

@lmmentel - our internal process for python is automated from pypi. Manually bringing in other source releases from GitHub release would require similar manual security / legal review, unfortunately. Having it on pypi makes it much easier both to get started, and when we want to bring in the latest releases.

[lmmentel]

Got it, thanks for the explanation.