Closed mehmetg closed 6 years ago
is this reproducible on the latest master? the error messages I see should be fixed by https://github.com/satori-com/satori-video-sdk-cpp/issues/101
I tried and it is not crashing for me. Another problem I see here is that we don't have enough laziness in our streams so it takes too long to generate the first packet, that's why it first tries to catch-up and drops a lot, but then it starts to work normally.
Please let me know if it is still reproducible
I just rebuilt lib error messages are gone. I am running longer experiments now and will report back if other problems come up.
Also, earlier drops in publisher mean that we may not be able to detect everything in those dew secs or mins right?
Thank you for the quick fix! 👍
yep, unfortunately, first minute is gone, that is also a bug, but at least it is not crashing :)
I need to investigate why it happens so let's keep this open
An issue seems to be there still, it takes a lot longer to manifest itself though. It has published 300K frames before crashing.
2017-11-23 16:45:37.253 (13360.083s) [main thread ] asio_streams_impl.h:320 WARN| late frame in interval
2017-11-23 16:45:37.254 (13360.084s) [input.encoded_b ] rtm_client.cpp:762 WARN| Forwarding request from thread input.encoded_b
2017-11-23 16:45:37.262 (13360.091s) [input.encoded_b ] rtm_client.cpp:762 WARN| Forwarding request from thread input.encoded_b
2017-11-23 16:45:37.263 (13360.093s) [input.encoded_b ] rtm_client.cpp:762 WARN| Forwarding request from thread input.encoded_b
2017-11-23 16:45:37.264 (13360.093s) [input.encoded_b ] rtm_client.cpp:762 WARN| Forwarding request from thread input.encoded_b
2017-11-23 16:45:37.265 (13360.094s) [input.encoded_b ] rtm_client.cpp:762 WARN| Forwarding request from thread input.encoded_b
2017-11-23 16:45:37.271 (13360.100s) [input.encoded_b ] rtm_client.cpp:762 WARN| Forwarding request from thread input.encoded_b
2017-11-23 16:45:37.272 (13360.102s) [input.encoded_b ] rtm_client.cpp:762 WARN| Forwarding request from thread input.encoded_b
2017-11-23 16:45:37.273 (13360.102s) [input.encoded_b ] rtm_client.cpp:762 WARN| Forwarding request from thread input.encoded_b
2017-11-23 16:45:37.274 (13360.103s) [input.encoded_b ] rtm_client.cpp:762 WARN| Forwarding request from thread input.encoded_b
2017-11-23 16:45:37.277 (13360.106s) [input.encoded_b ] rtm_client.cpp:762 WARN| Forwarding request from thread input.encoded_b
2017-11-23 16:45:37.278 (13360.107s) [input.encoded_b ] rtm_client.cpp:762 WARN| Forwarding request from thread input.encoded_b
2017-11-23 16:45:37.278 (13360.107s) [input.encoded_b ] rtm_client.cpp:762 WARN| Forwarding request from thread input.encoded_b
2017-11-23 16:45:37.279 (13360.108s) [input.encoded_b ] rtm_client.cpp:762 WARN| Forwarding request from thread input.encoded_b
2017-11-23 16:45:37.289 (13360.118s) [input.encoded_b ] rtm_client.cpp:762 WARN| Forwarding request from thread input.encoded_b
2017-11-23 16:45:37.289 (13360.118s) [input.encoded_b ] rtm_client.cpp:762 WARN| Forwarding request from thread input.encoded_b
2017-11-23 16:45:37.290 (13360.119s) [input.encoded_b ] rtm_client.cpp:762 WARN| Forwarding request from thread input.encoded_b
2017-11-23 16:45:37.290 (13360.119s) [input.encoded_b ] rtm_client.cpp:762 WARN| Forwarding request from thread input.encoded_b
2017-11-23 16:45:37.291 (13360.120s) [input.encoded_b ] rtm_client.cpp:762 WARN| Forwarding request from thread input.encoded_b
2017-11-23 16:45:37.293 (13360.122s) [input.encoded_b ] rtm_client.cpp:762 WARN| Forwarding request from thread input.encoded_b
2017-11-23 16:45:37.294 (13360.123s) [input.encoded_b ] rtm_client.cpp:762 WARN| Forwarding request from thread input.encoded_b
2017-11-23 16:45:37.294 (13360.124s) [input.encoded_b ] rtm_client.cpp:762 WARN| Forwarding request from thread input.encoded_b
2017-11-23 16:45:37.296 (13360.125s) [input.encoded_b ] rtm_client.cpp:762 WARN| Forwarding request from thread input.encoded_b
2017-11-23 16:45:37.297 (13360.126s) [input.encoded_b ] rtm_client.cpp:762 WARN| Forwarding request from thread input.encoded_b
libc++abi.dylib: terminating with uncaught exception of type boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::system::system_error> >: Broken pipe
Loguru caught a signal: SIGABRT
Stack trace:
23 0xb 27 ??? 0x000000000000000b 0x0 + 11
22 0x7fffae539235 start + 1
21 0x1069f31f9 main + 3289
20 0x1069f3aa1 boost::asio::io_service::run() + 49
19 0x106a00d04 boost::asio::detail::task_io_service::run(boost::system::error_code&) + 372
18 0x106a010f4 boost::asio::detail::task_io_service::do_run_one(boost::asio::detail::scoped_lock<boost::asio::detail::posix_mutex>&, boost::asio::detail::task_io_service_thread_info&, boost::system::error_code const&) + 500
17 0x106a01bd7 boost::asio::detail::task_io_service_operation::complete(boost::asio::detail::task_io_service&, boost::system::error_code const&, unsigned long) + 71
16 0x106bedb5b boost::asio::detail::completion_handler<satori::video::rtm::thread_checking_client::publish(std::string const&, cbor_item_t*, satori::video::rtm::publish_callbacks*)::$_1>::do_complete(boost::asio::detail::task_io_service*, boost::asio::detail::task_io_service_operation*, boost::system::error_code const&, unsigned long) + 171
15 0x106bedc97 void boost_asio_handler_invoke_helpers::invoke<satori::video::rtm::thread_checking_client::publish(std::string const&, cbor_item_t*, satori::video::rtm::publish_callbacks*)::$_1, satori::video::rtm::thread_checking_client::publish(std::string const&, cbor_item_t*, satori::video::rtm::publish_callbacks*)::$_1>(satori::video::rtm::thread_checking_client::publish(std::string const&, cbor_item_t*, satori::video::rtm::publish_callbacks*)::$_1&, satori::video::rtm::thread_checking_client::publish(std::string const&, cbor_item_t*, satori::video::rtm::publish_callbacks*)::$_1&) + 39
14 0x106bedd55 void boost::asio::asio_handler_invoke<satori::video::rtm::thread_checking_client::publish(std::string const&, cbor_item_t*, satori::video::rtm::publish_callbacks*)::$_1>(satori::video::rtm::thread_checking_client::publish(std::string const&, cbor_item_t*, satori::video::rtm::publish_callbacks*)::$_1&, ...) + 21
13 0x106beddc3 satori::video::rtm::thread_checking_client::publish(std::string const&, cbor_item_t*, satori::video::rtm::publish_callbacks*)::$_1::operator()() const + 99
12 0x106b33bd7 satori::video::rtm::resilient_client::publish(std::string const&, cbor_item_t*, satori::video::rtm::publish_callbacks*) + 759
11 0x106b3bc3c satori::video::rtm::(anonymous namespace)::secure_client::publish(std::string const&, cbor_item_t*, satori::video::rtm::publish_callbacks*) + 6540
10 0x106b4a0ee unsigned long boost::beast::websocket::stream<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> > > >::write<boost::asio::const_buffers_1>(boost::asio::const_buffers_1 const&) + 158
9 0x106b4eecb void boost::exception_detail::throw_exception_<boost::system::system_error>(boost::system::system_error const&, char const*, char const*, int) + 203
8 0x106b65c5b void boost::throw_exception<boost::exception_detail::error_info_injector<boost::system::system_error> >(boost::exception_detail::error_info_injector<boost::system::system_error> const&) + 123
7 0x7fffad1427be __cxxabiv1::exception_cleanup_func(_Unwind_Reason_Code, _Unwind_Exception*) + 0
6 0x7fffad142d49 std::__terminate(void (*)()) + 8
5 0x7fffadc55713 _objc_terminate() + 124
4 0x7fffad145c17 default_terminate_handler() + 243
3 0x7fffad12094a __cxa_bad_cast + 0
2 0x7fffae5cd420 abort + 129
1 0x4b743a240 5 ??? 0x00000004b743a240 0x0 + 20254532160
0 0x7fffae748b3a _sigtramp + 26
2017-11-23 16:45:38.205 (13361.034s) [main thread ] :0 FATL| Signal: SIGABRT
After #125, the publisher performs much better, but still segfaults on occassion:
2017-11-29 07:41:46.582 (3984.619s) [ C0EB6700] rtm_sink.cpp:49 0| published 99600 frames to 09-26-9-11
2017-11-29 07:41:50.582 (3988.619s) [ C0EB6700] rtm_sink.cpp:49 0| published 99700 frames to 09-26-9-11
2017-11-29 07:41:54.582 (3992.619s) [ C0EB6700] rtm_sink.cpp:49 0| published 99800 frames to 09-26-9-11
2017-11-29 07:41:58.583 (3996.619s) [ C0EB6700] rtm_sink.cpp:49 0| published 99900 frames to 09-26-9-11
2017-11-29 07:42:02.583 (4000.619s) [ C0EB6700] rtm_sink.cpp:49 0| published 100000 frames to 09-26-9-11
2017-11-29 07:42:06.582 (4004.619s) [ C0EB6700] rtm_sink.cpp:49 0| published 100100 frames to 09-26-9-11
2017-11-29 07:42:10.582 (4008.619s) [ C0EB6700] rtm_sink.cpp:49 0| published 100200 frames to 09-26-9-11
2017-11-29 07:42:14.583 (4012.619s) [ C0EB6700] rtm_sink.cpp:49 0| published 100300 frames to 09-26-9-11
2017-11-29 07:42:18.583 (4016.619s) [ C0EB6700] rtm_sink.cpp:49 0| published 100400 frames to 09-26-9-11
2017-11-29 07:42:22.582 (4020.619s) [ C0EB6700] rtm_sink.cpp:49 0| published 100500 frames to 09-26-9-11
2017-11-29 07:42:23.145 (4021.182s) [main thread ] rtm_client.cpp:408 ERR| asio error: sslv3 alert bad record mac
2017-11-29 07:42:23.145 (4021.182s) [main thread ] rtm_client.cpp:652 0| restarting rtm client because of error: asio error
Loguru caught a signal: SIGSEGV
Stack trace:
14 0x5642bc233a2a ./bin/satori_video_publisher_Linux(+0xb2a2a) [0x5642bc233a2a]
13 0x7ff0c150b830 __libc_start_main + 240
12 0x5642bc22ee54 ./bin/satori_video_publisher_Linux(+0xade54) [0x5642bc22ee54]
11 0x5642bc23e250 ./bin/satori_video_publisher_Linux(+0xbd250) [0x5642bc23e250]
10 0x5642bc28e04a ./bin/satori_video_publisher_Linux(+0x10d04a) [0x5642bc28e04a]
9 0x5642bc28dc5c ./bin/satori_video_publisher_Linux(+0x10cc5c) [0x5642bc28dc5c]
8 0x5642bc253556 ./bin/satori_video_publisher_Linux(+0xd2556) [0x5642bc253556]
7 0x5642bc28f09a ./bin/satori_video_publisher_Linux(+0x10e09a) [0x5642bc28f09a]
6 0x5642bc7df1b6 ./bin/satori_video_publisher_Linux(+0x65e1b6) [0x5642bc7df1b6]
5 0x5642bc7df08a ./bin/satori_video_publisher_Linux(+0x65e08a) [0x5642bc7df08a]
4 0x5642bcb754f1 malloc + 497
3 0x5642bcb44d65 ./bin/satori_video_publisher_Linux(+0x9c3d65) [0x5642bcb44d65]
2 0x5642bcb449a5 ./bin/satori_video_publisher_Linux(+0x9c39a5) [0x5642bcb449a5]
1 0x5642bcb44912 ./bin/satori_video_publisher_Linux(+0x9c3912) [0x5642bcb44912]
0 0x7ff0c216b390 /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390) [0x7ff0c216b390]
2017-11-29 07:42:23.146 (4021.182s) [main thread ] :0 FATL| Signal: SIGSEGV
./launch_rtm_live.sh: line 124: 21717 Segmentation fault (core dumped) ${VIDEO_PUBLISHER} --input-video-file ${VIDEO_FILE} --endpoint ${RTM_ENDPOINT} --appkey ${RTM_KEY} --channel ${CHANNEL}
Another error seen once the publisher reaches the end of the video file. It is harmless, mostly.
2017-11-30 01:55:30.040 (7194.101s) [ CC5E4700] rtm_client.cpp:275 0| Stopping secure RTM client
2017-11-30 01:55:30.040 (7194.102s) [main thread ] rtm_client.cpp:400 0| Got stop request for async_read loop
2017-11-30 01:55:30.040 (7194.102s) [main thread ] rtm_client.cpp:456 ERR| 0x555965d4c000 asio error: Bad file descriptor
2017-11-30 01:55:30.040 (7194.102s) [main thread ] rtm_client.cpp:652 0| restarting rtm client because of error: asio error
2017-11-30 01:55:30.041 (7194.102s) [main thread ] loguru.hpp:1637 0| atexit
Another error seen once the publisher reaches the end of the video file. It is harmless, mostly.
Judging by line numbers it doesn't look like the latest master.
After #125, the publisher performs much better, but still segfaults on occassion:
Also doesn't look like latest master.
Retested with the most recent master. Looks good has no issues in 10+ hrs of publishing.
The issue is not limited to large files even <1Gi files cause the issue, but it takes longer for reasons unknown to me.
File: https://drive.google.com/a/mz.com/file/d/1b4zFhJtInFS4VhEIwCKuU0QdveiO3t4x/view?usp=sharing
Command Line
Output