search

mozilla-services / megaphone

Firefox Global Broadcast API
Mozilla Public License 2.0
38 stars 12 forks source link

Invalid ROCKET_JSON_LOGGING #67

Closed glasserc closed 5 years ago

glasserc commented 5 years ago

Sometimes, but not always, our docker-compose cluster (which includes megaphone) doesn't come up properly. In the CI logs, I see errors like this:

thread 'main' panicked at 'rocket failed: ErrorMessage { msg: "Invalid ROCKET_JSON_LOGGING" }

stack backtrace:
   0: failure::backtrace::internal::InternalBacktrace::new::h24a4d3c627486c57 (0x562ba319d79f)
   1: <failure::backtrace::Backtrace as core::default::Default>::default::h2c347254eb0c26e0 (0x562ba319da2f)
   2: megaphone::logging::init_logging::h3833601807a02322 (0x562ba2f0c757)
   3: megaphone::http::rocket::h80dfefd9ef4e86bb (0x562ba2f4dd59)
   4: megaphone::main::hd69f0469f4276553 (0x562ba2f3f3f5)
   5: std::rt::lang_start::{{closure}}::ha845e59619a01779 (0x562ba2f07b72)
   6: std::rt::lang_start_internal::{{closure}}::h6780b53704329a7e (0x562ba31f2772)
             at src/libstd/rt.rs:49
      std::panicking::try::do_call::hc1f4a8d7f956e716
             at src/libstd/panicking.rs:297
   7: __rust_maybe_catch_panic (0x562ba31f7b09)
             at src/libpanic_unwind/lib.rs:92
   8: std::panicking::try::h3f2729e885829453 (0x562ba31f3385)
             at src/libstd/panicking.rs:276
      std::panic::catch_unwind::h390aa2e34eec2884
             at src/libstd/panic.rs:388
      std::rt::lang_start_internal::he0a86838f1552b68
             at src/libstd/rt.rs:48
   9: main (0x562ba2f3f4b1)
  10: __libc_start_main (0x7f7c70cec2e0)
  11: _start (0x562ba2f05839)
  12: <unknown> (0x0)', src/libcore/result.rs:999:5
stack backtrace:
   0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
             at src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:39
   1: std::sys_common::backtrace::_print
             at src/libstd/sys_common/backtrace.rs:70
   2: std::panicking::default_hook::{{closure}}
             at src/libstd/sys_common/backtrace.rs:58
             at src/libstd/panicking.rs:200
   3: std::panicking::default_hook
             at src/libstd/panicking.rs:215
   4: std::panicking::rust_panic_with_hook
             at src/libstd/panicking.rs:478
   5: std::panicking::continue_panic_fmt
             at src/libstd/panicking.rs:385
   6: rust_begin_unwind
             at src/libstd/panicking.rs:312
   7: core::panicking::panic_fmt
             at src/libcore/panicking.rs:85
   8: core::result::unwrap_failed
   9: megaphone::main
  10: std::rt::lang_start::{{closure}}
  11: std::panicking::try::do_call
             at src/libstd/rt.rs:49
             at src/libstd/panicking.rs:297
  12: __rust_maybe_catch_panic
             at src/libpanic_unwind/lib.rs:92
  13: std::rt::lang_start_internal
             at src/libstd/panicking.rs:276
             at src/libstd/panic.rs:388
             at src/libstd/rt.rs:48
  14: main
  15: __libc_start_main
  16: _start

Judging from the code, this should not happen if ROCKET_JSON_LOGGING is not set, but your Dockerfile doesn't set it, and neither does our docker-compose.yml. I also don't understand why this only happens sometimes -- it seems like we should hit this every time, but maybe logging is only being initialized lazily and we only see the failure if megaphone starts up before MySQL is ready for it?

pjenvey commented 5 years ago

Sorry about that, thanks Ethan!