Issue with deploying via EXRM releases

[mje113]

Things have been working really well with your awesome library--until I starting prepping for production releases. On Ubuntu, if I build a release via EXRM and then deploy the release to a different Ubuntu node, it looks like the databases aren't loading. However, if I deploy it to the node where the release was compiled, it works as expected.

The databases are in priv/ua_inspector and I've confirmed they make it into the release tarball.

config.exs:

config :ua_inspector,
  database_path: __DIR__
                 |> Path.dirname
                 |> Path.join("priv/ua_inspector")
                 |> Path.expand

If I open a remote connection to the "bad" node, it's clear the databases are empty:

Interactive Elixir (1.2.3) - press Ctrl+C to exit (type h() ENTER for help)
iex> UAInspector.Database.Clients.list
[]

Whereas the "good" node clearly is populated:

Interactive Elixir (1.2.3) - press Ctrl+C to exit (type h() ENTER for help)
iex> UAInspector.Database.Clients.list
[{1,
  %{engine: nil, name: "Akregator",
    regex: ~r/(?:^|[^A-Z0-9_-]|sprd-)(?:Akregator(?:\/(\d+[\.\d]+))?)/,
    type: "feed reader", version: "$1"}},
....

So, any ideas on what could potentially be different or how to better debug from my end?

Thanks!

[mneudert]

Could it be that the absolute path is not the same on both nodes?

Path.expand/1 makes it absolute. This would then get compiled into your package and not get reevaluated on the remote node. Then most likely resulting in the error.

Does it work if you just skip the path expansion?

[mje113]

Ah, thanks for the quick response!

Yep, that was it. I haven't found the right combination to work, however mapping it to a dir already on the server works, so I think I've got it from here.

[mje113]

While I was able to get things working in my prod environment by installing the databases manually and hard-coding their path in my config, I still haven't been able to find the right combo of building the databases directly in my exrm release.

I was about to take another stab at it, then I just saw the resolution on #3 and it made me wonder--would it be possible to inline the contents of all the yaml files directly at compile time? For my use-case at least, I would never be changing their contents without rolling out a new release. Doing that would alleviate the prod dependency on yaml parsing. If it's something you'd be interested I could spend a little time on PR.

[mneudert]

Yeah, code packaging is something not working really well with non-code-files at the moment.

But with inlining you have some manual work I would prefer not to impose on people. "Back in the days" the short codes were hardcoded. Every now and then the list got updated and I had to manually patch them. Not a useful solution :D

In the meantime I changed the loading process to reload all databases once a process crashes. That also suits the target of having a background updating process, for example :tzdata does that to update the database :timex uses.

Inlining the database might make the release process easier but having to recompile a module to update the database probably makes the other side worse.

Would it be a viable solution to just choose both ways at the same time? Like, following load process:

1. Load data from inline modules created during compilation. By default an empty database if the local database directory is empty as this definitely needs to work without relying on the availability of remote files.
2. Update the loaded database with locally available files. Triggered by a background agent in a production environment (or dev, or configured, ...).

That means inline compilation for (initial) deployment and then real-time updates through an optional background process.

Could that solve your needs while still paving the way for runtime updating the databases?

[mneudert]

Would you take the time to look at the branch precompiled-databases?

According to what you suggested I precompiled the database into a module. This data will be loaded first, after that the actual local file will trigger a reload if available.

Just a quick hack and only inlining the file contents of "bots.yml", not parsing them beforehand. But it might be a good starting point.

[mje113]

Ah, I get what you're saying about having databases get updated between deploys. Definitely supporting both having databases inlined as well as falling back to an updated version if present would work for my needs.

Let me take a poke at that branch.