search

chaseruskin / legoHDL

An experimental package manager and development tool for Hardware Description Languages (HDL).
https://c-rus.github.io/legoHDL
MIT License
14 stars 2 forks source link

Store released versions of designs as zip files in markets? #29

Closed chaseruskin closed 3 years ago

chaseruskin commented 3 years ago

Currently, markets rely on storing the git remote repository url for a given design to be able to clone it and check out the correct version number a user wishes to install. However, this comes with the idea that every design to be released must have a remote repository, and that remote repository must still exists beyond the release. These act as "pointers" to the design to be installed/downloaded. These "pointers" are a very lightweight single file that contain that url.

An alternative approach is to push compressed files (zip) to markets. These files will mean a not-so-tight integration with git is required (which is good, less hoops to jump through), as well as no remote repository is required for the design to be released to a market. This increases the reliability that if the zip file is seen, it can be installed. However, the tradeoff is that markets will now require considerably more space as each design release will be its own zip file.

Which approach should be used? Or should both be allowed and this design question be pushed off to the user-level?

chaseruskin commented 3 years ago

If the market is remote, all zip files won't need to be downloaded to user's machine, as git can fetch and pull only specific files (like a file that says what zip files are available). This would greatly cut down the size problem regarding using zip files as they would only live in the remote repository and not bog down a user's local machine. See https://stackoverflow.com/questions/16230838/is-it-possible-to-pull-just-one-file-in-git for specific file git pull approach.

chaseruskin commented 3 years ago

What about utilizing both methods? A hybrid approach: Store latest version of design as a zip (remember this is a git repository) along with its metadata file. Now, a design does not need to have a remote repository to be released to a market, and it is certain a user can download that block with the zip file available.

When needing a specific version, download the zip file and checkout the respective version from the git repository.

chaseruskin commented 3 years ago

Git does not handle binary files very well, taking up lots of storage for each different commit. The git repository data becomes large when adding a single zip and removing the older zip when releasing a block. A possible approach is Git LFS (https://git-lfs.github.com), but that becomes another dependency and forces markets to be a remote repository in order to work.

chaseruskin commented 3 years ago

Using git gc will perform optimizations on the repository, which may be helpful in cutting down storage as an immediate solution, if zip files are to be included in markets.

chaseruskin commented 3 years ago

Also, commits are not really necessary for markets because the latest commit has all available versions, through the version.log file and the .block zip file. Possibly keeping market repositories to only a certain number of commits will also cut down on excessive data storage. (https://stackoverflow.com/questions/1338728/delete-commits-from-a-branch-in-git)

chaseruskin commented 3 years ago

This issue is currently being closed as it is not important toward the initial PYPI release. If it eventually seems like a good idea, it can always be integrated in as future blocks get released and both approaches (either using the pointer file or downloading the zip) would be available to exist together. Progress can be seen on the release-with-zip branch.