Clean repository before publishing hex package

[badlop]

Luerl 1.1.0 and 1.1.1 hex packages include files that shouldn't be there. This can be seen, for example, in https://diff.hex.pm/diff/luerl/1.0.0..1.1.0

Some example files:

luerl_lib.erl-new
luerl_parse.erl.old
luerl_parse_exp.erl

And some of those files include references to paths that only existed in the original developer machine:

-file("/Users/rv/luerl/luerl/src/luerl_parse_exp.yrl", 187).
-file("/opt/homebrew/Cellar/erlang/26.2.1/lib/erlang/lib/parsetools-2.5/include/yeccpre.hrl

Compilation of Luerl 1.1.1 as a dependency using Erlang/OTP 20 fails due to those files. As a workaround, compilation succeeds if those files are removed using rebar3 clean before compiling.

Newer Erlang versions don't have this problem.

The solution? I don't have enough experience in hex packaging to be completely sure, but I hope this will be enough: for the next Luerl release, just before publishing the hex package, make sure the directory is clean of binaries, temporary or generated files.

[rvirding]

This just shows my ignorance about what is loaded up into hex using rebar3 hex. I thought it was basically took the same as went into github but it takes everything. I will fix that in the next release.

It will also contain the fix mentioned in your PR which I would like to change a bit. If this can be doe in the same PR that would be best.

[badlop]

Ok, I've read the documentation to find solutions to prevent those problems. And there are several interesting ideas:

As seen in https://hex.pm/docs/rebar3-publish

When running the command to publish a package, Hex will create a tar file of all the files and directories listed in the files property.

As luerl.app.src has "src" in files, then all the files you have in src/ when publishing will be included.

This means it is important to take care to only have relevant files before publishing. For example, rebar3 clean removes the files generated by rebar3.

But that isn't enough, as there may be editor temporary files (vim, emacs,...). I run this when I want to clean a git directory back to what is committed in git:

rm .gitignore
git clean -fd
git co .gitignore

---

Notice that, before the final publising, rebar3 shows what files are included, so you can review the file list. You can also use the --dry-run flag to ensure the package isn't yet published (documented in https://hexdocs.pm/rebar3_hex/rebar3_hex_publish.html )

For example:

$ rebar3 hex publish package --dry-run
===> Verifying dependencies...
Local Password:                         
===> package argument given, will not publish docs
===> luerl.app.src : deprecated field maintainers found
Publishing luerl 1.1.1 to hexpm
  Description: Luerl - an implementation of Lua on Erlang
  Dependencies:

  Included files:
    Emakefile
    LICENSE
    Makefile
    README.md
    src
    src/Elixir.Luerl.New.erl
    src/Elixir.Luerl.erl
    src/NOTES
    src/aaa_garbageeeeeeeeeeeeeeeee.txt
    ...
  Licenses: Apache-2.0
  Links:
    Github: https://github.com/rvirding/luerl
  Build tools: rebar3
Be aware, you are publishing to the public Hexpm repository.
Before publishing, please read Hex CoC: https://hex.pm/policies/codeofconduct
Proceed? ("Y")> 

Aha! this package contains a garbage file that I forgot in the src/ path!

And when you are happy with the package contents, you can proceed to the real publishing.

---

If you have problems numbering releases, git tagging, ... I found a nice feature that takes care to do it at the same time that publishing, rebar3 hex cut (documented in https://hexdocs.pm/rebar3_hex/rebar3_hex_cut.html ):

$ rebar3 hex cut
===> Verifying dependencies...
Select semver increment or other (Current 1.1.1):
1) patch
2) minor
3) major
4) other
[1-4] > 1

Create 'v1.1.2' commit? ("Y")> y

Push master to origin master? ("N")> n
===> No doc provider configuration was found, therefore docs can not be published.
...

Local Password:                         

===> luerl.app.src : deprecated field maintainers found
Publishing luerl 1.1.2 to hexpm
...
Proceed? ("Y")> n

Goodbye...
Create new git tag v1.1.2? ("Y")> y

---

By the way, in case a recently published package has some problem, it's possible to retire it https://hexdocs.pm/rebar3_hex/readme.html#retiring-packages

[rvirding]

So I have started by moving out all the files src that "git status" complains about. We will see if that is enough.

Where does "rebar3 hex" get its info from? I don't think I can retire the old one as it is six weeks old. I will have to fix a new one. When we get this with else and OTP27 fixed the unnecessary files removed then I will make a new version and push it to hex.

[badlop]

So I have started by moving out all the files src that "git status" complains about. We will see if that is enough.

Ok, but take into account that git status detects some of the files, but other files that are ignored by git (thanks to the file .gitignore) will be included in the hex package, and that shouldn't happen!

For example, the file luerl_parse.erl: it isn't detected by git, but hex detects it, includes it in the hex package...

Better to use git status --ignored=matching to check also the git-ignored files:

$ git status
Untracked files:
        src/aaa_garbage.erl

$ git status --ignored=matching
Untracked files:
        src/aaa_garbage.erl
Ignored files:
        erl_crash.dump
        rebar3.crashdump
        src/.DS_Store
        src/aaa_garbage.swp
        src/luerl_parse.erl
        src/luerl_scan.erl

$ rebar3 hex publish package --dry-run
    ...
    src/.DS_Store
    src/aaa_garbage.erl
    src/luerl_parse.erl

Where does "rebar3 hex" get its info from?

Looking at the documentation, it seems the list of files to include in a package is gotten from the files section in src/<myapp>.app.src.

And looking at the source code in rebar3_hex_build.erl it is obvious that it doesn't take into account gitignore, so it will include any files in src/ that match its criteria.

I don't think I can retire the old one as it is six weeks old. I will have to fix a new one.

Ok, no problem. I just mentioned it as a possibility. It seems useful a few minutes after publishing a package, and realizing it contains some major problem.

When we get this with else and OTP27 fixed the unnecessary files removed then I will make a new version and push it to hex.

That would be great! Luerl is used in ejabberd, and a new ejabberd release is planned for next week, so it would be great if it can include a new and improved Luerl version :star_struck:

[rvirding]

Ok, this has now been done and will be right in 1.2 which is coming RSN. It's already in develop.

[badlop]

Great, thanks!