Add support for ignore files

[aaronmallen]

I can see a scenario where developers may want files in their repository that do not need to be downloaded to the CC computer. As an example I like to minimize my lua files using npm and luamin, I'll typically develop in the src directory but really only need the dist directory downloaded to the cc machine. This could be implemented in a couple of ways

1. Like ruby gems you could specify files that should be included via a regex string or
2. You could specify which files should be ignored like a .gitignore maybe this could be implemented with a .packignore file.

I sort of like the second proposal there, but I'll leave that decision to you.

[lyqyd]

What about using a release branch in your repository containing only the files to be downloaded to the ComputerCraft machines, and specifying that branch for the package's download information?

[aaronmallen]

That's certainly one way to handle it, although I would say it forces the dev into a particular workflow... The way I got around it with the settings package is by passing the two files I needed as raw types and making one a dependency on the other. This method doesn't really scale well for larger projects...

[aaronmallen]

additionally by forcing a dev to use a release branch you're not quite solving the problem... I would need a special .gitignore on that branch which is a pain, and that .gitignore is still downloaded (trivial but still not a file that needs to be downloaded) additionally I now have to pick and choose if my README or my LICENSE should also live in the release branch even though neither of those is really needed.

[aaronmallen]

lastly: having an ignore file in a package manager is fairly common place .gitignore, .npmignore, etc... for a good reason.

[lyqyd]

Yes, a release branch does have the disadvantage of requiring a specific workflow if developers want to keep unrelated files in their git repo.

Git is not a package manager, and I see no evidence that ignore files are commonplace in package managers such as pacman, apt, etc., so this is not a reason to either add or omit an ignore file. I'm not sure there's an underlying "good reason" beyond "other design decisions require this for ease of use", which may, perhaps, end up applying here anyway.

Now, if this feature were to be implemented, it would add a bit of complexity. Instead of a download process like this:

1. Use the GitHub API to fetch the file listings
2. Fetch the file data for all files in the repository
3. Write the file data to disk

...we end up with this process:

1. Use the GitHub API to fetch the file listings
2. Check if one of those files is named .packignore
3. Download the .packignore file data if one exists
4. Parse the ignore file, removing all matching file listings for each line of the ignore file
5. Fetch the file data for any files remaining
6. Write the file data to disk

Obviously, the same would need to be done for BitBucket repos as well.

As an aside, there is the raw-multi download type, which was discussed in #39 and which I have apparently not fully documented yet.

I don't really want to solve this issue multiple ways, but this suggestion does have merit. I'll have to think about this one some more.

[aaronmallen]

Git is not a package manager

I would say this statement has low relevance to our discourse... correct it is not a package manager however most package managers (including your own) require it or some other version control, and a lot of them even utilize the .gitignore as their own ignore.

I would say a good chunk of non os based package management does allow for some form of file exclude/include functionality... npm, NuGet, bundler, etc...

I was unaware of a raw-multi package type, and am curious on how it was implemented and would love to see the docs for it.

I would take this a step further and say however this is implemented could be very closely tied to #27 as there you seem to have a need to include files outside of the traditional package tree. I haven't had a whole lot of time to go through your source yet, but am interested in helping anyway I can. I plan to deconstruct packman this weekend to better understand how it works :D

Finally:

Obviously, the same would need to be done for BitBucket repos as well.

I would say your suggested workflow isn't very DRY. Perhaps we automatically look for a .packignore if fs.isDir(package) or we allow it to be passed as an attribute on the packlist and we run this logic globally.

"Why the hell would I let you specify a .packignore in your packlist instead of just looking for it?" you ask... As a developer with MANY packages I am serving to packman I may want to utilize a global .packignore. In my case all of my projects have a .gitignore, a README, a LICENSE, a src folder, etc... and I would likely want to ignore everything but what's in my dist folder.

[lyqyd]

The implementation for multi, (not raw-multi, must've changed it when implementing) is pretty hackish. I'm not fully satisfied with it, and it's one of the reasons holding me back from implementing the help files portion of the package specifications. I'm still pondering where to go with it, but it's going to involve refactoring the package parsing.

The packignore file would never be created locally, we'd only download the data and parse it to remove entries from the list of files to be fetched before downloading the rest of the package. I'm not a fan of the idea of looking for these anywhere besides the root of the repository, since I don't want to look in two places for them, or require everyone to specify the URL if it's just in the repo anyway.

Please note that if you have files in a subfolder in your repository, that subfolder will also be created when using the github download method, so your files would appear in /usr/bin/dist/<name> with the default target. You'd have to use multi and specify each file if you wanted to not create that directory, or use a release branch as mentioned earlier.

[aaronmallen]

could this ALSO be worked around by doing type = raw and pointing it to an install script that pulls all the appropriate files down itself? Or is packman not going to like finding files it didn't install?

[lyqyd]

Packman exposes an "API" to the setup scripts, so that they can add files and folders to their package's list of files. If an install script does not make use of that API, the package won't be able to be removed unless you also use that script to clean up the files. It's best to just tell packman about the files if you're doing it that way. Telling packman about the files also means that the install script can remove itself after doing its job (of course, telling packman that it's being deleted). So, in short, yes, that would be another work-around.

The "API" that it exposes are these functions, and you can find them as the pack API, so pack.addFile, etc. The expected usage for the addFile call is to give it the desired path, then the contents to write to the file (usually just calling readAll() on the results of http.get), then the version of that file, if applicable. You can omit the version parameter if you're using this workaround, as packman won't be able to selectively update the files if you're not using the github or bitbucket download methods.

[aaronmallen]

is pack _G or do I need to call os.loadAPI("pack")?

[lyqyd]

It's placed in the environment of the script, so you use it as if it were in _G, no need to attempt to load it.

[aaronmallen]

is packman automatically exposing the programs to the root or is a user/package developer having to do that manually?

example if I install package foo can I immediately expect to be able to call foo args from the command line or am I expected to call usr/bin/foo args from the command line?

[lyqyd]

Packman itself doesn't modify the user's shell path string, but if you have the main/easy-shell package installed, it adds /usr/bin to the shell path. The main/easy-shell package is automatically installed when packman is installed via the pastebin setup script, but if a user sets up packman manually, they would have to modify the shell path themselves, or install main/easy-shell after installing packman. If a user chose to not do either of those options, they would have to use the full /usr/bin/foo path to run programs from that folder.

[aaronmallen]

Thanks for all the clarification man!