To host pre-built 3rd-party software to reduce install time

[hongquan]

When installing susi_linux for development, we need to install many dependencies. Some of them need to be compiled from C/C++ source. It takes long time, especially on weak device like Pi Zero. It wastes our developer's time when our product is still unstable and need to be reinstalled many time.

Solution: Build dependencies to *.deb and *.whl files (*.whl is Python wheel package, used for installing via pip) and host it somewhere. Those host can act as APT repo, PyPI to let us install via apt and pip.

Some services to consider:

• PackageCloud: Support APT, PyPI. Have pricing plan for OSS project. Normal free plan has limited quota.
• Bintray: Support APT but not PyPI. Have pricing plan for OSS project.
• Gemfury. Support APT, PyPI. Doesn't have pricing plan for OSS project but is free if all hosted packages are in public mode. Have to manually setup public mode for each file.

[hongquan]

In those of 3 services, I only have experience with PackageCloud (free plan) and Gemfury.

[mariobehling]

@hongquan Yes, sounds good. Which one do you want to go with. If no-one else has a preference, please help to set it up and let me know what else you need.

[hongquan]

I will go with PackageCloud. But to register our OSS project, have to contact them via email.

[hongquan]

@mariobehling The repo URL on PackageCloud will be in this form: https://packagecloud.io/[username]/[reponame], where I want reserve "fossasia" for "username". Can you create this account with some email address? My personal email address (ng.hong.quan@) is already bound with https://packagecloud.io/quan.

[hongquan]

@woshikie Please collect the cached .whl packages in your RPi Zero. We will upload it to repo later.

Note: Because pip save each .whl file to a subfolder inside ~/.cache/pip/wheels (e.g _.cache/pip/wheels/fc/cf/7c/3eb066f0a0dcb31b9ac6003aea79f3de116920bf3639460833/asyncpg-0.12.0-cp36-cp36m-linuxarmv7l.whl), it takes much effort to copy one by one. You should use find command to search for those file and copy at once.

Tip: I don't like the verbose syntax of find, so I use the alternative fd-find, which is written in Rust, and use it like this on my board:

$ cd /tmp/Whl/
$ fd armv7l ~/.cache/pip/wheels --exec cp {} ./

Explanation: fd will search for every file having armv7l in name, under subdirectories of ~/.cache/pip/wheels, then, with each result, it passes to {} and run cp {} ./, which is to copy the file to current folder.

For RPi Zero, which is ARMv6, you have to replace armv7l with something else.

Note 2: Poor you, there is no prebuilt binary of fd-find for RPi Zero yet.

[woshikie]

Looking into it

[hongquan]

I cannot wait for @mariobehling, so I temporarily create a repo here: https://repo.fury.io/fossasia/

Someone who want to upload your packages, please create account and give me your username, so that I add you as collaborator.

How to let pip install packages from this repo:

• Create pip configuration file.
• Add the URL https://repo.fury.io/fossasia/ as extra-index-url.

@stealthanthrax You can update the install.sh script, you copy this config file to /root/ and ~/ folders, before installing any Python packages. (@woshikie)

[mariobehling]

@hongquan I created an account at fury.io. Please add "mariobehling".

Packagecloud has is pretty pricy. So, does not seem suitable.

[hongquan]

PackageCloud is pricy, but it will be free for FOSS, @mariobehling .

I cannot change the avatar of the repo on Gemfury, because it depends on the admin email. Currently, I use my personal email for admin. I want to change it to some fossasia email.

[woshikie]

@hongquan I have also created an account. I am using the same username as github

[lskillen]

@hongquan Please feel free to consider Cloudsmith as well for packaging. :-)

It's also commercial but has very generous free plans and offerings for open-source projects (100GB of storage and 1TB of bandwidth, with more available on request). You can setup an org with several users to manage the account.

Full disclosure: I work for Cloudsmith, so might be biased (but only slightly, honest).

[hongquan]

@lskillen Great! I will give it a try. Thank you.