search

gotthardp / lorawan-server

Compact server for private LoRaWAN networks
https://gotthardp.github.io/lorawan-server
MIT License
946 stars 327 forks source link

Troubles upgrading from ver 0.5.3 to 0.6.2 #511

Open romansoft opened 5 years ago

romansoft commented 5 years ago

I tried upgrading from version 0.5.3 to 0.6.2 while preserving the Mnesia DB, but ran into problems related to MQTT connector:

2018-10-21 03:52:42 =SUPERVISOR REPORT====
     Supervisor: {local,lorawan_connector_sup}
     Context:    child_terminated
     Reason:     {{badmatch,[]},[{lorawan_application_backend,handle_downlink,2,[{file,"/home/travis/build/gotthardp/lorawan-server/_build/default/lib/lorawan_server/src/lorawan_application_backend.erl"},{line,178}]},{lorawan_connector_mqtt,handle_info,2,[{file,"/home/travis/build/gotthardp/lorawan-server/_build/default/lib/lorawan_server/src/lorawan_connector_mqtt.erl"},{line,217}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,637}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,711}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}
     Offender:   [{pid,<0.745.0>},{id,{mqtt,<<"ZMQTT">>}},{mfargs,{lorawan_connector_mqtt,start_link,[{connector,<<"ZMQTT">>,<<"SMP">>,<<"json">>,<<"mqtt://localhost:1883">>,undefined,<<"ZLoraOut">>,<<"ZLoraEvents">>,undefined,<<"ZLoraIn/#">>,<<"ZLoraIn">>,true,undefined,<<"gothard-lorawan-server">>,<<"normal">>,<<"user">>,<<"user">>,undefined,undefined,undefined,undefined,undefined,undefined}]}},{restart_type,transient},{shutdown,5000},{child_type,worker}]

So, I deleted the DB and let the service create a new one. Then, I imported the backup DB files (via dbimport). There was no error, but I'm not able to set Group for Gateways (it saves without error, but does not memorize the value). Profile tab shows no entries, and when trying to create one, it shows this error: The page you are looking for cannot be found. Take a break before trying again.

Also, I see the entry in crash.log.

2018-10-20 23:18:46 =ERROR REPORT====
Ranch listener http, connection process <0.389.0>, stream 2 had its request process <0.409.0> exit with reason badarg and stacktrace [{lorawan_http_digest,response,5,[{file,"/home/travis/build/gotthardp/lorawan-server/_build/default/lib/lorawan_server/src/lorawan_http_digest.erl"},{line,47}]},{lorawan_admin,handle_authentication,1,[{file,"/home/travis/build/gotthardp/lorawan-server/_build/default/lib/lorawan_server/src/lorawan_admin.erl"},{line,27}]},{lorawan_admin,handle_authorization_ex,2,[{file,"/home/travis/build/gotthardp/lorawan-server/_build/default/lib/lorawan_server/src/lorawan_admin.erl"},{line,56}]},{lorawan_admin_db_record,is_authorized,2,[{file,"/home/travis/build/gotthardp/lorawan-server/_build/default/lib/lorawan_server/src/lorawan_admin_db_record.erl"},{line,39}]},{cowboy_rest,call,3,[{file,"/home/travis/build/gotthardp/lorawan-server/_build/default/lib/cowboy/src/cowboy_rest.erl"},{line,1187}]},{cowboy_rest,is_authorized,2,[{file,"/home/travis/build/gotthardp/lorawan-server/_build/default/lib/cowboy/src/cowboy_rest.erl"},{line,350}]},{cowboy_rest,upgrade,4,[{file,"/home/travis/build/gotthardp/lorawan-server/_build/default/lib/cowboy/src/cowboy_rest.erl"},{line,264}]},{cowboy_stream_h,execute,3,[{file,"/home/travis/build/gotthardp/lorawan-server/_build/default/lib/cowboy/src/cowboy_stream_h.erl"},{line,293}]}]

(Just to let you know, Erlang was upgraded from ver 20 to 21 during the upgrade process)

romansoft commented 5 years ago

Here is an update after a long debug session.

There were multiple issues contributing:

  1. version 0.5.3 dbexport does not create profiles and network json backup files. I had to upgrade to 0.5.5 to be able to make a full database export
  2. Upgrading from 0.5.5 to 0.6.2 with preserved Mnesia database caused lorawan server crash 2018-10-21 10:55:35 =CRASH REPORT==== crasher: initial call: application_master:init/4 pid: <0.268.0> registered_name: [] exception exit: {{bad_return,{{lorawan_app,start,[normal,[]]},{'EXIT',{{badmatch,{aborted,{"Bad transform function",servers,#Fun<lorawan_db.15.46287602>,'lorawan@ip-172-31-27-175',{"Bad key or Record Name",{server,'lorawan@ip-172-31-27-175',[{{{2018,10,21},{15,53,59}},{0,0}},{{{2018,10,21},{15,52,59}},{0,0}},{{{2018,10,21},{15,51,59}},{0,0}},{{{2018,10,21},{15,50,59}},{0,0}},{{{2018,10,21},{15,49,59}},{0,0}},{{{2018,10,21},{15,48,59}},{0,0}},{{{2018,10,21},{15,47,59}},{0,0}},{{{2018,10,21},{15,46,59}},{0,0}},{{{2018,10,21},{15,45,59}},{0,0}},{{{2018,10,21},{15,44,59}},{0,0}},{{{2018,10,21},{15,43,59}},{0,0}},{{{2018,10,21},{15,42,59}},{0,0}},{{{2018,10,21},{15,41,59}},{0,0}},{{{2018,10,21},{15,40,59}},{0,0}},{{{2018,10,21},{15,39,59}},{0,0}}],undefined},{server,undefined,[{{{2018,10,21},{15,53,59}},{0,0}},{{{2018,10,21},{15,52,59}},{0,0}},{{{2018,10,21},{15,51,59}},{0,0}},{{{2018,10,21},{15,50,59}},{0,0}},{{{2018,10,21},{15,49,59}},{0,0}},{{{2018,10,21},{15,48,59}},{0,0}},{{{2018,10,21},{15,47,59}},{0,0}},{{{2018,10,21},{15,46,59}},{0,0}},{{{2018,10,21},{15,45,59}},{0,0}},{{{2018,10,21},{15,44,59}},{0,0}},{{{2018,10,21},{15,43,59}},{0,0}},{{{2018,10,21},{15,42,59}},{0,0}},{{{2018,10,21},{15,41,59}},{0,0}},{{{2018,10,21},{15,40,59}},{0,0}},{{{2018,10,21},{15,39,59}},{0,0}}]}}}}},[{lorawan_db,ensure_fields,2,[{file,"/home/travis/build/gotthardp/lorawan-server/_build/default/lib/lorawan_server/src/lorawan_db.erl"},{line,187}]},{lists,foreach,2,[{file,"lists.erl"},{line,1338}]},{lorawan_app,start,2,[{file,"/home/travis/build/gotthardp/lorawan-server/_build/default/lib/lorawan_server/src/lorawan_app.erl"},{line,17}]},{application_master,start_it_old,4,[{file,"application_master.erl"},{line,273}]}]}}}},[{application_master,init,4,[{file,"application_master.erl"},{line,134}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]} ancestors: [<0.267.0>] message_queue_len: 1 messages: [{'EXIT',<0.269.0>,normal}] links: [<0.267.0>,<0.33.0>] dictionary: [] trap_exit: true status: running heap_size: 2586 stack_size: 27 reductions: 290 neighbours: 2018-10-21 14:45:50 =ERROR REPORT==== ** Generic server <0.681.0> terminating ** Last message in was {publish,<<"ZLoraIn">>,<<"{\"devaddr\":\"0262AEF9\",\"port\":1,\"confirmed\":false,\"data\":\"015bcb68dd\"}">>} ** When Server state == {state,{connector,<<"ZMQTT">>,<<"SMP">>,<<"json">>,<<"mqtt://localhost:1883">>,undefined,<<"ZLoraOut">>,<<"ZLoraEvents">>,undefined,<<"ZLoraIn/#">>,<<"ZLoraIn">>,true,undefined,<<"gothard-lorawan-server">>,<<"normal">>,<<"user">>,<<"user">>,undefined,undefined,undefined,undefined,undefined,undefined},[{<<"mqtt://localhost:1883">>,[]},{<<"gothard-lorawan-server">>,[]},{<<"user">>,[]},{<<"user">>,[]}],{<<"ZLoraIn/#">>,[]},{<<"ZLoraOut">>,[]},{<<"ZLoraEvents">>,[]},{<<"ZLoraIn">>,[]},[{[<<"mqtt://localhost:1883">>,<<"gothard-lorawan-server">>,<<"user">>,<<"user">>],<0.682.0>,[<<"ZLoraIn/#">>],{costa,connected311,[{host,"localhost"},{port,1883},{logger,warning},{keepalive,0},{client_id,<<"gothard-lorawan-server">>},{username,<<"user">>},{password,<<"user">>}],{{2018,10,21},{19,45,50}},0}}]} ** Reason for termination == ** {{badmatch,[]},[{lorawan_application_backend,handle_downlink,2,[{file,"/home/travis/build/gotthardp/lorawan-server/_build/default/lib/lorawan_server/src/lorawan_application_backend.erl"},{line,178}]},{lorawan_connector_mqtt,handle_info,2,[{file,"/home/travis/build/gotthardp/lorawan-server/_build/default/lib/lorawan_server/src/lorawan_connector_mqtt.erl"},{line,217}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,616}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,686}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]} 2018-10-21 14:45:50 =CRASH REPORT==== crasher: initial call: lorawan_connector_mqtt:init/1 pid: <0.681.0> registered_name: [] I had to delete Mnesia database file, let lorawan server create a blank one, and dbimport backup DB exported from 0.5.5
  3. My web browser (Chrome) was caching the lorawan server session from the previous version and, for some reason, it was resulting in weird/misdirected/unsaved links to missing pages when I tried to update various configuration settings (ex. Profiles). When I opened the console in another browser, things started working correctly.
  4. In addition to creating Areas for Gateways and Groups for Profiles, as was described in upgrade instructions, I also had to add Scopes for different Users. Without Scopes, they had no default permissions to do anything.

To be able to debug the problem, I had to downgrade Erlang back to version 20.1. Now that my database migration is done to 6.0.2, is there any benefit to upgrading Erlang to version 21?

romansoft commented 5 years ago

In case someone runs into issue and they want to downgrade Erlang from the latest version to 20, here is how to do it:

sudo apt-get remove erlang-base erlang-crypto erlang-syntax-tools erlang-inets erlang-mnesia erlang-runtime-tools erlang-ssl erlang-public-key erlang-asn1 erlang-os-mon erlang-snmp erlang-xmerl

wget https://packages.erlang-solutions.com/erlang-solutions_1.0_all.deb
sudo dpkg -i erlang-solutions_1.0_all.deb
sudo apt-get update

sudo apt-get install erlang-base=1:20.1-1 erlang-crypto=1:20.1-1 erlang-syntax-tools=1:20.1-1 erlang-inets=1:20.1-1 erlang-mnesia=1:20.1-1 erlang-runtime-tools=1:20.1-1 erlang-ssl=1:20.1-1 erlang-public-key=1:20.1-1 erlang-asn1=1:20.1-1 erlang-os-mon=1:20.1-1 erlang-snmp=1:20.1-1 erlang-xmerl=1:20.1-1
gotthardp commented 5 years ago

Hello. Thanks for the debugging. I have few observations: 1) Yes, it's recommended to upgrade to latest 0.5.x before upgrading to 0.6.x. The dbexport scripts in previous releases were buggy. 2) The long CRASH report above seems to be generated by the 0.5.5 release (the line numbers don't match the 0.6.2), which is weird.

altishchenko commented 5 years ago

Also, it is a good idea to clear the browser cache, or reload a page with Shift-F5 - this is my solution to 'The page you are looking for cannot be found. Take a break before trying again.'

пн, 22 окт. 2018 г. в 18:57, Petr Gotthard notifications@github.com:

Hello. Thanks for the debugging. I have few observations:

  1. Yes, it's recommended to upgrade to latest 0.5.x before upgrading to 0.6.x. The dbexport scripts in previous releases were buggy.
  2. The long CRASH report above seems to be generated by the 0.5.5 release (the line numbers don't match the 0.6.2), which is weird.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/gotthardp/lorawan-server/issues/511#issuecomment-431877834, or mute the thread https://github.com/notifications/unsubscribe-auth/ASj_354kHIyGrk2rNUaVrUXdCZlQvuciks5uner8gaJpZM4XyRdj .

jot4p commented 5 years ago

THE MOST IMPORTANT THING IS CLEAR CACHE OF YOUR BROWSER :)

2 hours to trash with this.