search

vernemq / rebar3_cuttlefish

Cuttlefish plugin for rebar3
Other
26 stars 37 forks source link

override start_script/extended_start_script with a cuttlefish aware version #3

Closed marianoguerra closed 8 years ago

marianoguerra commented 9 years ago

hi, I was adapting extended_bin to be aware of cuttlefish and it makes sense to have it in this plugin, look in the overlay section if extended_start_script is set and override the start_script on bin with the cuttlefish aware version.

this would mean that this task should run as a post hook of release as mentioned in #1

tsloughter commented 9 years ago

That would be great, I'd like to make using cuttlefish with rebar3/relx as simple as possible and with as little manual intervention with overlays and such as possible.

marianoguerra commented 9 years ago

here is the latest version I'm using, it's in a template since some values are project dependent

https://github.com/marianoguerra/rebar3_template_riak_core/blob/master/rebar3_riak_core_extended_bin

how do you think we can handle this?

marianoguerra commented 9 years ago

ignore my last comment about the template since it's just copied, see here: https://github.com/marianoguerra/rebar3_template_riak_core/blob/master/rebar3_riak_core_extended_bin

it's used as a template in the overlay here: https://github.com/marianoguerra/rebar3_template_riak_core/blob/master/rebar3_riak_core_rebar.config.tpl#L35

but I think that part can be in this plugin?

tsloughter commented 9 years ago

Yea, that'd be cool as part of the plugin.

tsloughter commented 8 years ago

@marianoguerra I've completely redone the plugin, it handles everything now, let me know if it works for you.

marianoguerra commented 8 years ago

hi, I'm trying to make the little riak core code work against this version but I'm having (at least) one problem :D

releases/VSN/vm.args is not there, but releases/VSN/vm.args.orig is, do you know why/who set's it to vm.args.orig?

marianoguerra commented 8 years ago

I make it work, here is the commit:

https://github.com/marianoguerra/rebar3_template_riak_core/commit/37ea0b8614a50fc548f0f747080c3c3c6d1fbed6

notes:

tsloughter commented 8 years ago

I'll be making it work with rebar3 release in the next release.

If you update your rebar3 to master you shouldn't get the vm.args.orig file anymore with that.

You should have always had to have an override with rebar3, unless something changed with riak_ensemble. Also, I forked it https://github.com/tsloughter/riak_ensemble and got rid of that clock because I'm only supporting 18+ and published to hex https://hex.pm/packages/riak_ensemble

marianoguerra commented 8 years ago

I'm on latest master and still get the vm.args.orig,

you can replicate it with:

mkdir -p ~/.config/rebar3/templates
git clone https://github.com/marianoguerra/rebar3_template_riak_core.git ~/.config/rebar3/templates/rebar3_riak_core
rebar3 new rebar3_riak_core ricor
cd ricor
make release
ls _build/default/rel/ricor/releases/0.1.0

I get:

ricor.boot  ricor.rel  ricor.script  start_clean.boot  sys.config.orig  vm.args.orig

rebar version:

$ rebar3 report
Rebar3 report
 version alpha-4+build.3305.refe3437c1
 generated at 2016-02-21T15:56:49+00:00
=================
Please submit this along with your issue at https://github.com/rebar/rebar3/issues (and feel free to edit out private information, if any)
-----------------
Task: 
Entered as:

-----------------
Operating System: x86_64-pc-linux-gnu
ERTS: Erlang/OTP 18 [erts-7.0] [source] [64-bit] [smp:4:4] [async-threads:0] [kernel-poll:false]
Root Directory: /usr/lib/erlang
Library directory: /usr/lib/erlang/lib
-----------------
Loaded Applications:
bbmustache: 1.0.4
certifi: 0.3.0
cf: 0.2.1
common_test: 1.11
compiler: 6.0
crypto: 3.6
cth_readable: 1.2.0
dialyzer: 2.8
edoc: 0.7.17
erlware_commons: 0.19.0
eunit: 2.2.10
eunit_formatters: 0.3.1
getopt: 0.8.2
inets: 6.0
kernel: 4.0
providers: 1.6.0
public_key: 1.0
relx: 3.17.0
sasl: 2.5
snmp: 5.2
ssl_verify_hostname: 1.0.5
stdlib: 2.5
syntax_tools: 1.7
tools: 2.8

-----------------
Escript path: /home/mariano/bin/rebar3
Providers:
  app_discovery as clean compile compile cover ct deps dialyzer do edoc escriptize eunit help install install_deps list lock new path pkgs release release relup report run shell tar tar tree unlock update upgrade upgrade upgrade version xref
tsloughter commented 8 years ago

Oh, I think I put the overrides in the wrong order :).

Issue is you shouldn't have {sys_config, "config/sys.config"} or {vm_args, "config/vm.args"} but it won't matter with my latest fix because the cuttlefish release provider forces them to be false even if you have them set.

marianoguerra commented 8 years ago

I tried it again, build seems to be working, I have some questions:

also, when trying to start it I'm getting a crash that looks like bad command line argument parsing, I get this error:

$ ./_build/default/rel/ricor/bin/ricor console                         
Exec: /usr/lib/erlang/erts-7.0/bin/erlexec -boot /home/mariano/tmp/ricor/_build/default/rel/ricor/releases/0.1.0/ricor -boot_var ERTS_LIB_DIR /usr/lib/erlang/erts-7.0/../lib  -mode embedded  -config /home/mariano/tmp/ricor/_build/default/rel/ricor/generated/app.2016.02.23.08.55.22.config -args_file /home/mariano/tmp/ricor/_build/default/rel/ricor/generated/vm.2016.02.23.08.55.22.args -vm_args /home/mariano/tmp/ricor/_build/default/rel/ricor/generated/vm.2016.02.23.08.55.22.args  -- console
Root: /home/mariano/tmp/ricor/_build/default/rel/ricor
{error_logger,{{2016,2,23},{8,55,23}},"Error in process ~p with exit value:~n~p~n",[<0.510.0>,{{case_clause,{'EXIT',{function_clause,[{application_controller,'-check_conf/0-fun-0-',[["/home/mariano/tmp/ricor/_build/default/rel/ricor/generated/app.2016.02.23.08.55.22.config","ERL_FULLSWEEP_AFTER","0"],[]],[{file,"application_controller.erl"},{line,1809}]},{lists,foldl,3,[{file,"lists.erl"},{line,1262}]},{application_controller,check_conf,0,[{file,"application_controller.erl"},{line,1808}]},{application_controller,init,2,[{file,"application_controller.erl"},{line,485}]}]}}},[{application_controller,init,2,[{file,"application_controller.erl"},{line,485}]}]}]}
{"could not start kernel pid",application_controller,"{{case_clause,{'EXIT',{function_clause,[{application_controller,'-check_conf/0-fun-0-',[[\"/home/mariano/tmp/ricor/_build/default/rel/ricor/generated/app.2016.02.23.08.55.22.config\",\"ERL_FULLSWEEP_AFTER\",\"0\"],[]],[{file,\"application_controller.erl\"},{line,1809}]},{lists,foldl,3,[{file,\"lists.erl\"},{line,1262}]},{application_controller,check_conf,0,[{file,\"application_controller.erl\"},{line,1808}]},{application_controller,init,2,[{file,\"application_controller.erl\"},{line,485}]}]}}},[{application_controller,init,2,[{file,\"application_controller.erl\"},{line,485}]}]}"}

Crash dump is being written to: -env...done
could not start kernel pid (application_controller) ({{case_clause,{'EXIT',{function_clause,[{application_controller,'-check_conf/0-fun-0-',[["/home/mariano/tmp/ricor/_build/default/rel/ricor/genera

as you can see (or not in that mess :D) it fails because it calls check_conf with this list of config files: 

[["/home/mariano/tmp/ricor/_build/default/rel/ricor/generated/app.2016.02.23.08.55.22.config","ERL_FULLSWEEP_AFTER","0"],

clearly "ERL_FULLSWEEP_AFTER" and "0" aren't config files, you can find them in the generated vm.args file, which has this content:

+P 256000                                                                      
-env ERL_MAX_ETS_TABLES 256000                                                 
-env ERL_CRASH_DUMP                                                            
-env ERL_FULLSWEEP_AFTER 0                                                     
-env ERL_MAX_PORTS 262144                                                      
+A 64                                                                          
-setcookie erlang                                                              
-name ricor@127.0.0.1                                                          
+K true                                                                        
+W w                                                                           
-smp enable

another thing that points at bad command line argument parsing is this line:

Crash dump is being written to: -env...done

the option to dump the files is "-env" which clearly is a case of flags as values, and now that I write this I notice that

-env ERL_CRASH_DUMP

doesn't have a value

as an aside, during schema discovery, do you avoid files like this? 

./_build/default/lib/cuttlefish/test/bad_erlang.schema

the ERL_CRASH_DUMP flag comes from _build/default/plugins/cuttlefish/priv/erlang_vm.schema which has this configuration:

{mapping, "erlang.crash_dump", "vm_args.-env ERL_CRASH_DUMP", [                
  {default, ""},                                                               
  {datatype, file},                                                            
  hidden                                                                       
]}.

the default is the empty string, which when inserted in vm.args it messes the command line argument order (I think)

the solution I think would be to either add a better default to the crash_dump option or more generally, detect options that require a value here https://github.com/basho/cuttlefish/blob/develop/src/cuttlefish_escript.erl#L394 and remove them if the value is the empty string.

what do you think?

PS: adding erlang.crash_dump = erlang_crash.dump to my etc/ricor.conf made it boot

marianoguerra commented 8 years ago

(I'm reusing this issue because it's all part of the same, if you prefer different issues for each let me know)

when running:

rebar3 as dev1 release

I get:

===> Unable to copy from /home/mariano/tmp/ricor/_build/dev1/bin/cuttlefish to /home/mariano/tmp/ricor/_build/dev1/rel/ricor/bin/cuttlefish because of {copy_failed, enoent}

This is what was being talked about yesterday on IRC

should I change it to something else?

should cuttlefish be built and copied for each profile I have?

in case we want to keep the profile on the operations, isn't it safer to copy it from _build/{{profile}}/lib/cuttlefish/cuttlefish?

tsloughter commented 8 years ago

Ah crap. Yea, I think we need to change this back. @Licenser was right with his patch until we moved to project_plugins which are going to always be in default :)

marianoguerra commented 8 years ago

@tsloughter what's your take on the cuttlefish empty default? should I open an issue on cuttlefish? maybe there's a temporary fix until we get a cuttlefish release without that problem?

tsloughter commented 8 years ago

I don't know. I just put an entry. But would be nice if the default was erl_crash.dump

marianoguerra commented 8 years ago

put an entry where? manually in the config? or in your plugin?

just asking to know what to do on my projects and templates

tsloughter commented 8 years ago

In vars.config for overlay_vars, at least for me erlang_vm.schema has {default, "{{crash_dump}}"}

marianoguerra commented 8 years ago

great! thanks for the tip.

when https://github.com/tsloughter/rebar3_cuttlefish/issues/3#issuecomment-187600950 is solved we can close this issue and #7

tsloughter commented 8 years ago

@marianoguerra I think it should be resolved with my most recent commit to master.

marianoguerra commented 8 years ago

no filename:join/3? :)

https://github.com/tsloughter/rebar3_cuttlefish/blob/master/src/rebar3_cuttlefish_release.erl#L40

marianoguerra commented 8 years ago

doing the _checkouts trick and replacing those 3 args for a list seems to make the plugin work fully for me.

marianoguerra commented 8 years ago

latest master works :)