Spec out future of LFE project deployments and responsibilities for lfeool

oubiwann commented 9 years ago

Introduction

There have been several discussions in emails, mail lists, bugs, PRs, at conferences, etc., about:

the responsibilities of LFE and lfetool
the best way to get started on LFE quickly (for new users)
how to manage dependencies
configuration, releases, and deployment of LFE applications

Robert has been considering ways in which LFE might support a "core" set of libraries that would be part of a standard install of LFE. lfetool has recently been updated with bootstrapping capabilities which will allow it to use more and more native LFE code (as opposed to bash), leading to a gradual rewrite in LFE. Both of these efforts have been considering the use of configuration files and standard locations for LFE libraries.

As we approach a 1.0 release for LFE and as lfetool prepares to incorporate more LFE into its growing code base, it's time to lay out a clear plan and define some standards for both of these as well as for the wider LFE community so that new and old users alike are aware of the preferred (and supported!) way of working with LFE in development and production environments.

This Ticket

This ticket will be a living document that attempts to work out a standard approach and development plan for these features. This ticket will very likely spawn many other new tickets in LFE, lfetool, and other tools/projects/libraries. These will be linked here.

This ticket will also keep track of tasks lists and TODO items.

Foundation

Use Cases

Download LFE and get started as quickly as possible
Use LFE with an "approved" set of libraries, supported by LFE out of the box (without the need for other tools)
Create a new LFE-based library, script, application, service, web app, etc.
Performing LFE-based releases
...
Conceptual Categorization of Tasks
Managing code:paths
Managing downloads of dependencies
Allowing for different versions of libraries/dependencies to be downloaded and used
Establishing best practices for LFE and the core/approved libraries it bootstraps/uses/allows to be used out of the box
Establishing best practices for LFE projects and applications with regard to dependencies, code paths, library inclusion, etc.
Unifying configuration across all tools into an LFE-syntax config file
...
Robert's Ideas on LFE Installation

@rvirding recently distilled his thoughts on this topic in an email, which I will present here for continued reference:

Build the core LFE as it comes from me. With this you can start LFE using the lfe/lfec/lfescript commands and also use it when compiling LFE applications. But it only contains the core system without any added libraries.
Make a base "package" which contains the core LFE plus a number of "standard" libraries. This should have the structure of erlang release where all the parts are erlang applications kept in the 'lib' directory. It is a release but uses the existing erlang with all its applications. This will also allow you to run LFE with the lfe/lfec/lfescript commands but give you automatic access to the LFE libraries. With this structure it is easy to add new LFE applications as they become available.
Build a self-contained release containing LFE core, specific LFE libraries (applications), and all other applications from other places that are needed. This is for making a release which can be shipped as a unit.
Planning

Todo Items
- [ ] Experiment with relx and see what the workflows for an LFE project look like
- [ ] Walk through this tutorial with an LFE app: https://erlangcentral.org/topic/erlang-mk-and-relx/
- [ ] Maybe this one too: http://ninenines.eu/articles/erlang.mk-and-relx/
- [ ] Document LFE relx usage based on the steps taken when doing the above tutorials
- [ ] See if we can forgo the ERL_LIBS wrangling of lfetool and use Erlang releases instead
- [ ] See if it might make sense for LFE to expand upon the default set of library paths Erlang defines (e.g., adding "LFE-approved library paths via (code:add_pathsa <dirs>))
- [ ] Explore the feasibility of an LFE repository of dependencies (e.g., ~/.lfe/deps/*)
- [ ] See if it's possible with the current state of tools to have multiple installations of libraries with different versions
- [ ] Experiment with LFE stdlib (see https://github.com/rvirding/lfe/issues/107)
- [ ] Simple idea for a first iteration of an LFE stdlb (see https://github.com/rvirding/lfe/issues/108)

oubiwann commented 9 years ago

I've had some problems with relx (it's not actually building on my machine right now), so progress here is a little bit slow (balancing with other exciting tasks, too ;-))

oubiwann commented 9 years ago

Turned out my rebar was too old; I've just build relx using the latest rebar release.

arpunk commented 9 years ago

@oubiwann I'm currently using relx to generate my releases and tarballs. The process can be quite simple:

1) Ensure you have the proper dependencies specified. In your relx.conf:

{release, {your-app, "0.0.1"}, [lfe, sasl, logjam, lager, ..., your-app]}.
{extended_start_script, true}.
{overlay, [{template, "lfe.config", "lfe.config"}]}.
{vm_args, "./etc/vm.args"}.

I put my configs under your-app/etc and I trust logjam to create my your-app/logs directory, I just need to make sure lfe.config is on the correct place in order to correctly start the application.

2) Generate the release:

$ relx release

3) And release tarball aswell:

$ relx tar

That's it, now you can upload your release (should be on $PWD/_rel/your-app/your-app-0.0.1.tar.gz). On production systems I'm able to attach and run an LFE shell like a normal relx release:

$ your-app/bin/your-app start
$ your-app/bin/your-app attach
...
> lfe_shell:start().

I'm not sure if I'm missing something but until now everything has been working fine.

rvirding commented 9 years ago

A quick question: does this create a completely stand-alone release which contains everything including the erlang VM? I am not relx knowledgeable. A follow-on question would be: how do I create an LFE release containing all the LFE applications I want but uses the existing erlang system?

Robert

On 11 February 2015 at 14:46, Ricardo Lanziano notifications@github.com wrote:

@oubiwann https://github.com/oubiwann I'm currently using relx to generate my releases and tarballs. The process can be quite simple:

1) Ensure you have the proper dependencies specified. In your relx.conf:

{release, {your-app, "0.0.1"}, [lfe, sasl, logjam, lager, ..., your-app]}. {extended_start_script, true}. {overlay, [{template, "lfe.config", "lfe.config"}]}. {vm_args, "./etc/vm.args"}.

I put my configs under your-app/etc and I trust logjam to create my your-app/logs directory, I just need to make sure lfe.config is on the correct place in order to correctly start the application.

2) Generate the release:

$ relx release

3) And release tarball aswell:

$ relx tar

That's it, now you can upload your release (should be on $PWD/_rel/your-app/your-app-0.0.1.tar.gz). On production systems I'm able to attach and run an LFE shell like a normal relx release:

$ your-app/bin/your-app start $ your-app/bin/your-app attach ...

lfe_shell:start().

I'm not sure if I'm missing something but until now everything has been working fine.

— Reply to this email directly or view it on GitHub https://github.com/lfe/lfetool/issues/162#issuecomment-73882743.

arpunk commented 9 years ago

@rvirding According to the documentation, by default relx will include local ERTS but one can specify the contrary on the relx.conf as:

{include_erts, false}.

This will create a release without the local ERTS but with the required libraries for running the LFE application (including LFE).

lfe-deprecated / lfetool