search

mopidy / mopidy-spotify

Mopidy extension for playing music from Spotify
https://mopidy.com/ext/spotify/
Apache License 2.0
936 stars 109 forks source link

mopidy run in docker on armhf hangs with 100% CPU usage #402

Open mczerski opened 2 months ago

mczerski commented 2 months ago

I have a armhf armbian bullseye system with mopidy in docker. It was WORKING with gst-plugin-spotify v0.9.1. After the issue described here happend I had to upgrade gst-plugin-spotify to v0.13.1. Now it is NOT WORKING. I can go to web interface, select spotify song to play and hit play, but after that mopidy process goes 100% CPU and nothing else happens. Anyone has the same issue ? I created the minimal deockerfile (below): Dockerfile:

FROM arm32v7/debian:bullseye-slim

RUN apt update && \
    apt install -y \
        mopidy \
        python3-pip

RUN pip install \
    mopidy \
    mopidy-spotify==5.0.0a2 \
    Mopidy-MusicBox-Webclient

COPY gst-plugin-spotify_0.13.1-1_armhf.deb /
RUN dpkg -i /gst-plugin-spotify_0.13.1-1_armhf.deb

COPY mopidy.conf /etc/mopidy/
RUN mkdir -p /var/lib/mopidy/spotify/credentials-cache/
COPY credentials.json /var/lib/mopidy/spotify/credentials-cache/

CMD ["mopidy", "--config", "/etc/mopidy/mopidy.conf"]

mopidy.conf:

[http]
enabled = true
hostname = 0.0.0.0

[spotify]
enabled = true
username = XXX
password = XXX
client_id = XXX
client_secret = XXX
allow_cache = false
docker build --network=host -t test-mopidy-spotify .
docker run -d --rm --network host --name test-mopidy-spotify test-mopidy-spotify
kingosticks commented 2 months ago

Can you provide a log ?

mczerski commented 2 months ago

nothing important in mopidy logs. In gstreamer logs, with GST_DEBUG=3,spotify:6, after playing the song:

0:00:28.190274119     8  0x15607a0 DEBUG        spotifyaudiosrc audio/spotify/src/spotifyaudiosrc/imp.rs:354:<gstspotify::spotifyaudiosrc::imp::SpotifyAudioSrc as gstreamer::subclass::uri_handler::URIHandlerImpl>::set_uri:<source> set URI: spotify:track:71fSn6OIF7HnCbgCzOBqpj
0:00:28.197646613     8 0xf1605380 DEBUG        spotifyaudiosrc audio/spotify/src/common.rs:144:gstspotify::common::Settings::connect_session::{{closure}}:<source> reuse cached credentials for user xzgfvdc9r165xst3txjd0zzua
0:00:28.199231912     8  0x14e4370 WARN                    alsa conf.c:5200:snd_config_expand: alsalib error: Unknown parameters {AES0 0x02 AES1 0x82 AES2 0x00 AES3 0x02}
0:00:28.199311787     8  0x14e4370 WARN                    alsa pcm.c:2660:snd_pcm_open_noupdate: alsalib error: Unknown PCM default:{AES0 0x02 AES1 0x82 AES2 0x00 AES3 0x02}
mczerski commented 2 months ago

also i'm watching cpu usage with htop. it shows that most of the usage is RED - Kernel threads

kingosticks commented 2 months ago

And non-spotify tracks don't have this issue? Do they still have the alsa param errors?

mczerski commented 2 months ago

local files are working and produce such log

0:00:19.831573348     8  0x10fa7a0 WARN                 basesrc gstbasesrc.c:3688:gst_base_src_start_complete:<source> pad not activated yet
0:00:19.837045208     8  0x10fa7a0 WARN                 basesrc gstbasesrc.c:3688:gst_base_src_start_complete:<source> pad not activated yet
0:00:19.849541519     8  0x107e370 WARN                    alsa conf.c:5200:snd_config_expand: alsalib error: Unknown parameters {AES0 0x02 AES1 0x82 AES2 0x00 AES3 0x02}
0:00:19.849691269     8  0x107e370 WARN                    alsa pcm.c:2660:snd_pcm_open_noupdate: alsalib error: Unknown PCM default:{AES0 0x02 AES1 0x82 AES2 0x00 AES3 0x02}
0:00:19.869566657     8 0xf2a056c0 FIXME                  id3v2 gstid3tag.c:141:gst_tag_from_id3_tag: Cannot map ID3v2 tag 'TCMP' to GStreamer tag
0:00:19.869673616     8 0xf2a056c0 FIXME                  id3v2 gstid3tag.c:141:gst_tag_from_id3_tag: Cannot map ID3v2 tag 'TMED' to GStreamer tag
0:00:19.869730491     8 0xf2a056c0 FIXME                  id3v2 gstid3tag.c:141:gst_tag_from_id3_tag: Cannot map ID3v2 tag 'TLAN' to GStreamer tag
0:00:19.869759492     8 0xf2a056c0 FIXME                  id3v2 gstid3tag.c:141:gst_tag_from_id3_tag: Cannot map ID3v2 tag 'TDOR' to GStreamer tag
0:00:19.869785325     8 0xf2a056c0 FIXME                  id3v2 gstid3tag.c:141:gst_tag_from_id3_tag: Cannot map ID3v2 tag 'USLT' to GStreamer tag
0:00:19.869937492     8 0xf2a056c0 FIXME                  id3v2 gstid3tag.c:141:gst_tag_from_id3_tag: Cannot map ID3v2 tag 'TIPL' to GStreamer tag
0:00:19.869970909     8 0xf2a056c0 FIXME                  id3v2 gstid3tag.c:141:gst_tag_from_id3_tag: Cannot map ID3v2 tag 'TXXX' to GStreamer tag
0:00:19.870002826     8 0xf2a056c0 FIXME                  id3v2 gstid3tag.c:213:gst_tag_from_id3_user_tag: Cannot map ID3v2 user tag 'Script' of type 'TXXX' to GStreamer tag
0:00:19.870073868     8 0xf2a056c0 FIXME                  id3v2 gstid3tag.c:141:gst_tag_from_id3_tag: Cannot map ID3v2 tag 'TXXX' to GStreamer tag
0:00:19.870097076     8 0xf2a056c0 FIXME                  id3v2 gstid3tag.c:213:gst_tag_from_id3_user_tag: Cannot map ID3v2 user tag 'ASIN' of type 'TXXX' to GStreamer tag
0:00:19.870124993     8 0xf2a056c0 FIXME                  id3v2 gstid3tag.c:141:gst_tag_from_id3_tag: Cannot map ID3v2 tag 'TXXX' to GStreamer tag
0:00:19.870192910     8 0xf2a056c0 FIXME                  id3v2 gstid3tag.c:213:gst_tag_from_id3_user_tag: Cannot map ID3v2 user tag 'Artist Credit' of type 'TXXX' to GStreamer tag
0:00:19.870246827     8 0xf2a056c0 FIXME                  id3v2 gstid3tag.c:141:gst_tag_from_id3_tag: Cannot map ID3v2 tag 'TXXX' to GStreamer tag
0:00:19.870269994     8 0xf2a056c0 FIXME                  id3v2 gstid3tag.c:213:gst_tag_from_id3_user_tag: Cannot map ID3v2 user tag 'ALBUMARTISTSORT' of type 'TXXX' to GStreamer tag
0:00:19.870435495     8 0xf2a056c0 FIXME                  id3v2 gstid3tag.c:141:gst_tag_from_id3_tag: Cannot map ID3v2 tag 'TXXX' to GStreamer tag
0:00:19.870463912     8 0xf2a056c0 FIXME                  id3v2 gstid3tag.c:213:gst_tag_from_id3_user_tag: Cannot map ID3v2 user tag 'Album Artist Credit' of type 'TXXX' to GStreamer tag
0:00:19.870514287     8 0xf2a056c0 FIXME                  id3v2 gstid3tag.c:141:gst_tag_from_id3_tag: Cannot map ID3v2 tag 'TXXX' to GStreamer tag
0:00:19.870538662     8 0xf2a056c0 FIXME                  id3v2 gstid3tag.c:213:gst_tag_from_id3_user_tag: Cannot map ID3v2 user tag 'MusicBrainz Album Status' of type 'TXXX' to GStreamer tag
0:00:19.870589746     8 0xf2a056c0 FIXME                  id3v2 gstid3tag.c:141:gst_tag_from_id3_tag: Cannot map ID3v2 tag 'TXXX' to GStreamer tag
0:00:19.870614912     8 0xf2a056c0 FIXME                  id3v2 gstid3tag.c:213:gst_tag_from_id3_user_tag: Cannot map ID3v2 user tag 'MusicBrainz Album Type' of type 'TXXX' to GStreamer tag
0:00:19.870644996     8 0xf2a056c0 FIXME                  id3v2 gstid3tag.c:141:gst_tag_from_id3_tag: Cannot map ID3v2 tag 'TXXX' to GStreamer tag
0:00:19.870667454     8 0xf2a056c0 FIXME                  id3v2 gstid3tag.c:213:gst_tag_from_id3_user_tag: Cannot map ID3v2 user tag 'MusicBrainz Album Release Country' of type 'TXXX' to GStreamer tag
0:00:19.870770746     8 0xf2a056c0 FIXME                  id3v2 gstid3tag.c:141:gst_tag_from_id3_tag: Cannot map ID3v2 tag 'TXXX' to GStreamer tag
0:00:19.870937414     8 0xf2a056c0 FIXME                  id3v2 gstid3tag.c:141:gst_tag_from_id3_tag: Cannot map ID3v2 tag 'UFID' to GStreamer tag
0:00:19.870981706     8 0xf2a056c0 FIXME                  id3v2 gstid3tag.c:141:gst_tag_from_id3_tag: Cannot map ID3v2 tag 'TXXX' to GStreamer tag
0:00:19.871013748     8 0xf2a056c0 FIXME                  id3v2 gstid3tag.c:141:gst_tag_from_id3_tag: Cannot map ID3v2 tag 'TXXX' to GStreamer tag
0:00:19.871043873     8 0xf2a056c0 FIXME                  id3v2 gstid3tag.c:141:gst_tag_from_id3_tag: Cannot map ID3v2 tag 'TXXX' to GStreamer tag
0:00:19.871075373     8 0xf2a056c0 FIXME                  id3v2 gstid3tag.c:141:gst_tag_from_id3_tag: Cannot map ID3v2 tag 'TXXX' to GStreamer tag
0:00:19.935132641     8 0xf2009598 WARN         audio-resampler audio-resampler.c:274:convert_taps_gint16_c: can't find exact taps
0:00:19.941701465     8 0xf2a5cc58 WARN                    alsa pcm_hw.c:1359:snd_pcm_hw_get_chmap: alsalib error: Cannot read Channel Map ctl
mczerski commented 1 month ago

I added htop and gdb to docker image to track down what is going on, while the CPU goes 100% this is the state: htop: mopidy_spotify_htop gdb:


(gdb) c
Continuing.
^C
Thread 1 "mopidy" received signal SIGINT, Interrupt.
[Switching to Thread 0xf71ae310 (LWP 1)]
__libc_do_syscall () at ../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:46
46  in ../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S
(gdb) info thread
  Id   Target Id                                    Frame 
* 1    Thread 0xf71ae310 (LWP 1) "mopidy"           __libc_do_syscall () at ../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:46
  2    Thread 0xf47ba460 (LWP 10) "gmain"           __libc_do_syscall () at ../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:46
  3    Thread 0xf3dff460 (LWP 11) "mopidy"          __libc_do_syscall () at ../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:46
  4    Thread 0xf33ff460 (LWP 12) "mopidy"          __libc_do_syscall () at ../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:46
  5    Thread 0xf29ff460 (LWP 13) "mopidy"          __libc_do_syscall () at ../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:46
  6    Thread 0xf1fff460 (LWP 14) "mopidy"          __libc_do_syscall () at ../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:46
  7    Thread 0xf15ff460 (LWP 15) "mopidy"          __libc_do_syscall () at ../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:46
  8    Thread 0xf0bff460 (LWP 16) "mopidy"          __libc_do_syscall () at ../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:46
  9    Thread 0xef7fe460 (LWP 24) "mopidy"          __libc_do_syscall () at ../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:46
  10   Thread 0xeeffd460 (LWP 25) "mopidy"          __libc_do_syscall () at ../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:46
  11   Thread 0xee7fc460 (LWP 26) "mopidy"          __libc_do_syscall () at ../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:47
  14   Thread 0xeffff460 (LWP 44) "mopidy"          0xed64d908 in ?? () from /usr/lib/arm-linux-gnueabihf/gstreamer-1.0/libgstspotify.so
  15   Thread 0xed431460 (LWP 45) "source:src"      __libc_do_syscall () at ../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:46
  16   Thread 0xecc30460 (LWP 46) "tokio-runtime-w" __libc_do_syscall () at ../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:47
  17   Thread 0xec9d6460 (LWP 47) "source:src"      syscall () at ../sysdeps/unix/sysv/linux/arm/syscall.S:37
(gdb) thread 14
[Switching to thread 14 (Thread 0xeffff460 (LWP 44))]
#0  0xed64d908 in ?? () from /usr/lib/arm-linux-gnueabihf/gstreamer-1.0/libgstspotify.so
(gdb) bt
#0  0xed64d908 in ?? () from /usr/lib/arm-linux-gnueabihf/gstreamer-1.0/libgstspotify.so
#1  0x00000000 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb) 

``
mczerski commented 1 month ago

I can reproduce the issue with gst-launch command.

First something that works (http stream):

# gst-launch-1.0 playbin uri=http://mp3.polskieradio.pl:8900/
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
Got context from element 'source': gst.soup.session=context, session=(SoupSession)NULL, force=(boolean)false;
Redistribute latency...
Pipeline is PREROLLED ...
Prerolled, waiting for buffering to finish...
Setting pipeline to PLAYING ...
New clock: GstAudioSinkClock

and spotify that hangs with 100% CPU usage:

# gst-launch-1.0 playbin uri=spotify:track:3i3P1mGpV9eRlfKccjDjwi?username=$USERNAME\&password=$PASSWORD
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...

On the other armhf system, without docker:

# gst-launch-1.0 playbin uri=spotify:track:3i3P1mGpV9eRlfKccjDjwi?username=$USERNAME\&password=$PASSWORD
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
ERROR: from element /GstPlayBin:playbin0/GstURIDecodeBin:uridecodebin0/GstSpotifyAudioSrc:source: Could not get/set settings from/on resource.
Additional debug info:
audio/spotify/src/spotifyaudiosrc/imp.rs(280): <gstspotify::spotifyaudiosrc::imp::SpotifyAudioSrc as gstreamer_base::subclass::push_src::PushSrcImpl>::create (): /GstPlayBin:playbin0/GstURIDecodeBin:uridecodebin0/GstSpotifyAudioSrc:source:
Permission denied { Login failed with reason: Bad credentials }
ERROR: pipeline doesn't want to preroll.
Setting pipeline to NULL ...
ERROR: from element /GstPlayBin:playbin0/GstURIDecodeBin:uridecodebin0/GstSpotifyAudioSrc:source: Internal data stream error.
Additional debug info:
../libs/gst/base/gstbasesrc.c(3127): gst_base_src_loop (): /GstPlayBin:playbin0/GstURIDecodeBin:uridecodebin0/GstSpotifyAudioSrc:source:
streaming stopped, reason error (-5)
ERROR: pipeline doesn't want to preroll.
Freeing pipeline ...

This does not hang and fails with credentials issue as expected.

mczerski commented 1 month ago

and log with GST_DEBUG=5: log.txt

after launching the command this is the part of log that keeps repeating endlessly:

0:00:00.405050467 15195 0xac544200 DEBUG                  query gstquery.c:679:gst_query_new_custom: creating new query 0xac611120 position
0:00:00.405152633 15195 0xac544200 DEBUG       GST_ELEMENT_PADS gstelement.c:2051:gst_element_query: send query on element playbin0
0:00:00.405217675 15195 0xac544200 DEBUG                    bin gstbin.c:4337:gst_bin_query:<playbin0> Sending query 0xac611120 (type position) to sink children
0:00:00.405304300 15195 0xac544200 DEBUG             GST_STATES gstbin.c:1996:bin_element_is_sink:<playbin0> child uridecodebin0 is not sink
0:00:00.405341549 15195 0xac544200 DEBUG             GST_STATES gstbin.c:1996:bin_element_is_sink:<playbin0> child playsink is sink
0:00:00.405371383 15195 0xac544200 DEBUG       GST_ELEMENT_PADS gstelement.c:2051:gst_element_query: send query on element playsink
0:00:00.405408883 15195 0xac544200 DEBUG                    bin gstbin.c:4337:gst_bin_query:<playsink> Sending query 0xac611120 (type position) to sink children
0:00:00.405441424 15195 0xac544200 DEBUG             GST_STATES gstbin.c:1996:bin_element_is_sink:<playsink> child streamsynchronizer0 is not sink
0:00:00.405479841 15195 0xac544200 DEBUG                    bin gstbin.c:4360:gst_bin_query:<playsink> query 0xac611120 result 0
0:00:00.405518049 15195 0xac544200 DEBUG                    bin gstbin.c:4360:gst_bin_query:<playbin0> query 0xac611120 result 0
mczerski commented 1 month ago

I built gst-plugin-spotify v.0.13.2 in debug version. started with rust-gdb and this is what I get after starting gst-launch commad.

(gdb) info thread
  Id   Target Id                                     Frame 
* 1    Thread 0xf77e15f0 (LWP 101) "gst-launch-1.0"  __libc_do_syscall () at ../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:46
  2    Thread 0xf5e66440 (LWP 104) "gst-launch-1.0"  syscall () at ../sysdeps/unix/sysv/linux/arm/syscall.S:37
  3    Thread 0xf5aff440 (LWP 105) "source:src"      __libc_do_syscall () at ../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:46
  4    Thread 0xf52fe440 (LWP 106) "tokio-runtime-w" 0xf63266d0 in core::sync::atomic::atomic_compare_exchange_weak<usize> (dst=0xf5b075b8, old=9, new=<optimized out>, 
    success=core::sync::atomic::Ordering::AcqRel, failure=core::sync::atomic::Ordering::Acquire) at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/sync/atomic.rs:3430
  5    Thread 0xf4eff440 (LWP 107) "gmain"           __libc_do_syscall () at ../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:46
  6    Thread 0xf44ff440 (LWP 108) "tokio-runtime-w" syscall () at ../sysdeps/unix/sysv/linux/arm/syscall.S:37
  7    Thread 0xf3eff440 (LWP 109) "source:src"      syscall () at ../sysdeps/unix/sysv/linux/arm/syscall.S:37
(gdb) thread 4
[Switching to thread 4 (Thread 0xf52fe440 (LWP 106))]
#0  0xf63266d0 in core::sync::atomic::atomic_compare_exchange_weak<usize> (dst=0xf5b075b8, old=9, new=<optimized out>, success=core::sync::atomic::Ordering::AcqRel, 
    failure=core::sync::atomic::Ordering::Acquire) at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/sync/atomic.rs:3430
3430    /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/sync/atomic.rs: No such file or directory.
(gdb) bt
#0  0xf63266d0 in core::sync::atomic::atomic_compare_exchange_weak<usize> (dst=0xf5b075b8, old=9, new=<optimized out>, success=core::sync::atomic::Ordering::AcqRel, 
    failure=core::sync::atomic::Ordering::Acquire) at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/sync/atomic.rs:3430
#1  core::sync::atomic::AtomicUsize::compare_exchange_weak (self=0xf5b075b8, current=9, new=<optimized out>, success=core::sync::atomic::Ordering::AcqRel, failure=core::sync::atomic::Ordering::Acquire)
    at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/sync/atomic.rs:2642
#2  tokio::sync::oneshot::State::set_complete (cell=0xf5b075b8) at src/sync/oneshot.rs:1337
#3  0xf6304f14 in tokio::sync::oneshot::Inner<core::result::Result<http::response::Response<hyper::body::body::Body>, (hyper::error::Error, core::option::Option<http::request::Request<hyper::body::body::Body>>)>>::complete<core::result::Result<http::response::Response<hyper::body::body::Body>, (hyper::error::Error, core::option::Option<http::request::Request<hyper::body::body::Body>>)>> (
    self=0xf5b074f8) at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/sync/oneshot.rs:1126
#4  tokio::sync::oneshot::Sender<core::result::Result<http::response::Response<hyper::body::body::Body>, (hyper::error::Error, core::option::Option<http::request::Request<hyper::body::body::Body>>)>>::send<core::result::Result<http::response::Response<hyper::body::body::Body>, (hyper::error::Error, core::option::Option<http::request::Request<hyper::body::body::Body>>)>> (self=..., t=...)
    at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/sync/oneshot.rs:609
#5  0xf62fbb14 in hyper::client::dispatch::Callback<http::request::Request<hyper::body::body::Body>, http::response::Response<hyper::body::body::Body>>::send<http::request::Request<hyper::body::body::Body>, http::response::Response<hyper::body::body::Body>> (self=..., val=...) at src/client/dispatch.rs:256
#6  0xf61bf7ac in hyper::proto::h1::dispatch::{impl#5}::recv_msg<hyper::body::body::Body> (self=<optimized out>, msg=...)
    at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.30/src/proto/h1/dispatch.rs:614
#7  0xf61c280c in hyper::proto::h1::dispatch::Dispatcher<hyper::proto::h1::dispatch::Client<hyper::body::body::Body>, hyper::body::body::Body, tokio::net::tcp::stream::TcpStream, hyper::proto::h1::role::Client>::poll_read_head<hyper::proto::h1::dispatch::Client<hyper::body::body::Body>, hyper::body::body::Body, tokio::net::tcp::stream::TcpStream, hyper::proto::h1::role::Client> (self=0xf5b071f0, 
    cx=<optimized out>) at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.30/src/proto/h1/dispatch.rs:270
#8  0xf61c40c8 in hyper::proto::h1::dispatch::Dispatcher<hyper::proto::h1::dispatch::Client<hyper::body::body::Body>, hyper::body::body::Body, tokio::net::tcp::stream::TcpStream, hyper::proto::h1::role::Client>::poll_read<hyper::proto::h1::dispatch::Client<hyper::body::body::Body>, hyper::body::body::Body, tokio::net::tcp::stream::TcpStream, hyper::proto::h1::role::Client> (self=0xf5b071f0, 
    cx=<optimized out>) at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.30/src/proto/h1/dispatch.rs:191
#9  0xf61c33a8 in hyper::proto::h1::dispatch::Dispatcher<hyper::proto::h1::dispatch::Client<hyper::body::body::Body>, hyper::body::body::Body, tokio::net::tcp::stream::TcpStream, hyper::proto::h1::role::Client>::poll_loop<hyper::proto::h1::dispatch::Client<hyper::body::body::Body>, hyper::body::body::Body, tokio::net::tcp::stream::TcpStream, hyper::proto::h1::role::Client> (self=0xf5b071f0, 
    cx=0xf52fd948) at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.30/src/proto/h1/dispatch.rs:163
#10 0xf61c06ac in hyper::proto::h1::dispatch::Dispatcher<hyper::proto::h1::dispatch::Client<hyper::body::body::Body>, hyper::body::body::Body, tokio::net::tcp::stream::TcpStream, hyper::proto::h1::role::Client>::poll_inner<hyper::proto::h1::dispatch::Client<hyper::body::body::Body>, hyper::body::body::Body, tokio::net::tcp::stream::TcpStream, hyper::proto::h1::role::Client> (self=0xf5b071f0, 
    cx=0xf52fd948, should_shutdown=true) at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.30/src/proto/h1/dispatch.rs:140
#11 0xf61bfeac in hyper::proto::h1::dispatch::Dispatcher<hyper::proto::h1::dispatch::Client<hyper::body::body::Body>, hyper::body::body::Body, tokio::net::tcp::stream::TcpStream, hyper::proto::h1::role::Client>::poll_catch<hyper::proto::h1::dispatch::Client<hyper::body::body::Body>, hyper::body::body::Body, tokio::net::tcp::stream::TcpStream, hyper::proto::h1::role::Client> (self=0xf5b071f0, cx=0xb, 
    should_shutdown=true) at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.30/src/proto/h1/dispatch.rs:119
#12 0xf61e7b48 in hyper::proto::h1::dispatch::{impl#1}::poll<hyper::proto::h1::dispatch::Client<hyper::body::body::Body>, hyper::body::body::Body, tokio::net::tcp::stream::TcpStream, hyper::proto::h1::role::Client> (self=..., cx=0xf52fd948) at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.30/src/proto/h1/dispatch.rs:435
#13 hyper::client::conn::{impl#10}::poll<tokio::net::tcp::stream::TcpStream, hyper::body::body::Body> (cx=0xf52fd948, self=...)
    at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.30/src/client/conn.rs:1108
#14 hyper::client::conn::{impl#5}::poll<tokio::net::tcp::stream::TcpStream, hyper::body::body::Body> (self=..., cx=0xf52fd948)
    at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.30/src/client/conn.rs:559
#15 0xf61a8d8c in futures_core::future::{impl#2}::try_poll<hyper::client::conn::Connection<tokio::net::tcp::stream::TcpStream, hyper::body::body::Body>, (), hyper::error::Error> (self=..., cx=0x9)
    at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.30/src/future.rs:82
#16 futures_util::future::try_future::into_future::{impl#2}::poll<hyper::client::conn::Connection<tokio::net::tcp::stream::TcpStream, hyper::body::body::Body>> (self=..., cx=0x9)
    at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/try_future/into_future.rs:34
#17 futures_util::future::future::map::{impl#2}::poll<futures_util::future::try_future::into_future::IntoFuture<hyper::client::conn::Connection<tokio::net::tcp::stream::TcpStream, hyper::body::body::Body>>, futures_util::fns::MapErrFn<hyper::client::client::{impl#3}::connect_to::{closure#0}::{closure#0}::{async_block#0}::{closure_env#0}<hyper::client::connect::http::HttpConnector<hyper::client::connect::dns::GaiResolver>, hyper::body::body::Body>>, core::result::Result<(), ()>> (self=..., cx=0x9)
    at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/future/map.rs:55
#18 0xf61a8b30 in futures_util::future::future::map::{impl#2}::poll<futures_util::future::try_future::MapErr<hyper::client::conn::Connection<tokio::net::tcp::stream::TcpStream, hyper::body::body::Body>, hyper::client::client::{impl#3}::connect_to::{closure#0}::{closure#0}::{async_block#0}::{closure_env#0}<hyper::client::connect::http::HttpConnector<hyper::client::connect::dns::GaiResolver>, hyper::body::body::Body>>, hyper::client::client::{impl#3}::connect_to::{closure#0}::{closure#0}::{async_block#0}::{closure_env#1}<hyper::client::connect::http::HttpConnector<hyper::client::connect::dns::GaiResolver>, hyper::body::body::Body>, ()> (self=..., cx=0x9) at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/future/map.rs:55
#19 0xf61ca500 in tokio::runtime::task::core::{impl#6}::poll::{closure#0}<futures_util::future::future::Map<futures_util::future::try_future::MapErr<hyper::client::conn::Connection<tokio::net::tcp::stream::TcpStream, hyper::body::body::Body>, hyper::client::client::{impl#3}::connect_to::{closure#0}::{closure#0}::{async_block#0}::{closure_env#0}<hyper::client::connect::http::HttpConnector<hyper::client::connect::dns::GaiResolver>, hyper::body::body::Body>>, hyper::client::client::{impl#3}::connect_to::{closure#0}::{closure#0}::{async_block#0}::{closure_env#1}<hyper::client::connect::http::HttpConnector<hyper::client::connect::dns::GaiResolver>, hyper::body::body::Body>>, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle, alloc::alloc::Global>> (ptr=0xf5b071e8)
    at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/core.rs:331
#20 tokio::loom::std::unsafe_cell::UnsafeCell<tokio::runtime::task::core::Stage<futures_util::future::future::Map<futures_util::future::try_future::MapErr<hyper::client::conn::Connection<tokio::net::tcp:--Type <RET> for more, q to quit, c to continue without paging--
:stream::TcpStream, hyper::body::body::Body>, hyper::client::client::{impl#3}::connect_to::{closure#0}::{closure#0}::{async_block#0}::{closure_env#0}<hyper::client::connect::http::HttpConnector<hyper::client::connect::dns::GaiResolver>, hyper::body::body::Body>>, hyper::client::client::{impl#3}::connect_to::{closure#0}::{closure#0}::{async_block#0}::{closure_env#1}<hyper::client::connect::http::HttpConnector<hyper::client::connect::dns::GaiResolver>, hyper::body::body::Body>>>>::with_mut<tokio::runtime::task::core::Stage<futures_util::future::future::Map<futures_util::future::try_future::MapErr<hyper::client::conn::Connection<tokio::net::tcp::stream::TcpStream, hyper::body::body::Body>, hyper::client::client::{impl#3}::connect_to::{closure#0}::{closure#0}::{async_block#0}::{closure_env#0}<hyper::client::connect::http::HttpConnector<hyper::client::connect::dns::GaiResolver>, hyper::body::body::Body>>, hyper::client::client::{impl#3}::connect_to::{closure#0}::{closure#0}::{async_block#0}::{closure_env#1}<hyper::client::connect::http::HttpConnector<hyper::client::connect::dns::GaiResolver>, hyper::body::body::Body>>>, core::task::poll::Poll<()>, tokio::runtime::task::core::{impl#6}::poll::{closure_env#0}<futures_util::future::future::Map<futures_util::future::try_future::MapErr<hyper::client::conn::Connection<tokio::net::tcp::stream::TcpStream, hyper::body::body::Body>, hyper::client::client::{impl#3}::connect_to::{closure#0}::{closure#0}::{async_block#0}::{closure_env#0}<hyper::client::connect::http::HttpConnector<hyper::client::connect::dns::GaiResolver>, hyper::body::body::Body>>, hyper::client::client::{impl#3}::connect_to::{closure#0}::{closure#0}::{async_block#0}::{closure_env#1}<hyper::client::connect::http::HttpConnector<hyper::client::connect::dns::GaiResolver>, hyper::body::body::Body>>, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle, alloc::alloc::Global>>> (self=0xf5b071e8, f=...)
    at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/loom/std/unsafe_cell.rs:16
#21 tokio::runtime::task::core::Core<futures_util::future::future::Map<futures_util::future::try_future::MapErr<hyper::client::conn::Connection<tokio::net::tcp::stream::TcpStream, hyper::body::body::Body>, hyper::client::client::{impl#3}::connect_to::{closure#0}::{closure#0}::{async_block#0}::{closure_env#0}<hyper::client::connect::http::HttpConnector<hyper::client::connect::dns::GaiResolver>, hyper::body::body::Body>>, hyper::client::client::{impl#3}::connect_to::{closure#0}::{closure#0}::{async_block#0}::{closure_env#1}<hyper::client::connect::http::HttpConnector<hyper::client::connect::dns::GaiResolver>, hyper::body::body::Body>>, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle, alloc::alloc::Global>>::poll<futures_util::future::future::Map<futures_util::future::try_future::MapErr<hyper::client::conn::Connection<tokio::net::tcp::stream::TcpStream, hyper::body::body::Body>, hyper::client::client::{impl#3}::connect_to::{closure#0}::{closure#0}::{async_block#0}::{closure_env#0}<hyper::client::connect::http::HttpConnector<hyper::client::connect::dns::GaiResolver>, hyper::body::body::Body>>, hyper::client::client::{impl#3}::connect_to::{closure#0}::{closure#0}::{async_block#0}::{closure_env#1}<hyper::client::connect::http::HttpConnector<hyper::client::connect::dns::GaiResolver>, hyper::body::body::Body>>, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle, alloc::alloc::Global>> (self=0xf5b071d8, cx=...) at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/core.rs:320
#22 0xf61d48a8 in tokio::runtime::task::harness::poll_future::{closure#0}<futures_util::future::future::Map<futures_util::future::try_future::MapErr<hyper::client::conn::Connection<tokio::net::tcp::stream::TcpStream, hyper::body::body::Body>, hyper::client::client::{impl#3}::connect_to::{closure#0}::{closure#0}::{async_block#0}::{closure_env#0}<hyper::client::connect::http::HttpConnector<hyper::client::connect::dns::GaiResolver>, hyper::body::body::Body>>, hyper::client::client::{impl#3}::connect_to::{closure#0}::{closure#0}::{async_block#0}::{closure_env#1}<hyper::client::connect::http::HttpConnector<hyper::client::connect::dns::GaiResolver>, hyper::body::body::Body>>, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle, alloc::alloc::Global>> ()
    at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/harness.rs:500
#23 core::panic::unwind_safe::{impl#23}::call_once<core::task::poll::Poll<()>, tokio::runtime::task::harness::poll_future::{closure_env#0}<futures_util::future::future::Map<futures_util::future::try_future::MapErr<hyper::client::conn::Connection<tokio::net::tcp::stream::TcpStream, hyper::body::body::Body>, hyper::client::client::{impl#3}::connect_to::{closure#0}::{closure#0}::{async_block#0}::{closure_env#0}<hyper::client::connect::http::HttpConnector<hyper::client::connect::dns::GaiResolver>, hyper::body::body::Body>>, hyper::client::client::{impl#3}::connect_to::{closure#0}::{closure#0}::{async_block#0}::{closure_env#1}<hyper::client::connect::http::HttpConnector<hyper::client::connect::dns::GaiResolver>, hyper::body::body::Body>>, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle, alloc::alloc::Global>>> (self=...) at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/panic/unwind_safe.rs:272
#24 std::panicking::try::do_call<core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future::{closure_env#0}<futures_util::future::future::Map<futures_util::future::try_future::MapErr<hyper::client::conn::Connection<tokio::net::tcp::stream::TcpStream, hyper::body::body::Body>, hyper::client::client::{impl#3}::connect_to::{closure#0}::{closure#0}::{async_block#0}::{closure_env#0}<hyper::client::connect::http::HttpConnector<hyper::client::connect::dns::GaiResolver>, hyper::body::body::Body>>, hyper::client::client::{impl#3}::connect_to::{closure#0}::{closure#0}::{async_block#0}::{closure_env#1}<hyper::client::connect::http::HttpConnector<hyper::client::connect::dns::GaiResolver>, hyper::body::body::Body>>, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle, alloc::alloc::Global>>>, core::task::poll::Poll<()>> (data=<optimized out>) at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs:557
#25 std::panicking::try<core::task::poll::Poll<()>, core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future::{closure_env#0}<futures_util::future::future::Map<futures_util::future::try_future::MapErr<hyper::client::conn::Connection<tokio::net::tcp::stream::TcpStream, hyper::body::body::Body>, hyper::client::client::{impl#3}::connect_to::{closure#0}::{closure#0}::{async_block#0}::{closure_env#0}<hyper::client::connect::http::HttpConnector<hyper::client::connect::dns::GaiResolver>, hyper::body::body::Body>>, hyper::client::client::{impl#3}::connect_to::{closure#0}::{closure#0}::{async_block#0}::{closure_env#1}<hyper::client::connect::http::HttpConnector<hyper::client::connect::dns::GaiResolver>, hyper::body::body::Body>>, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle, alloc::alloc::Global>>>> (f=<error reading variable: Cannot access memory at address 0xc>) at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs:521
#26 std::panic::catch_unwind<core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future::{closure_env#0}<futures_util::future::future::Map<futures_util::future::try_future::MapErr<hyper::client::conn::Connection<tokio::net::tcp::stream::TcpStream, hyper::body::body::Body>, hyper::client::client::{impl#3}::connect_to::{closure#0}::{closure#0}::{async_block#0}::{closure_env#0}<hyper::client::connect::http::HttpConnector<hyper::client::connect::dns::GaiResolver>, hyper::body::body::Body>>, hyper::client::client::{impl#3}::connect_to::{closure#0}::{closure#0}::{async_block#0}::{closure_env#1}<hyper::client::connect::http::HttpConnector<hyper::client::connect::dns::GaiResolver>, hyper::body::body::Body>>, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle, alloc::alloc::Global>>>, core::task::poll::Poll<()>> (f=...) at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panic.rs:350
#27 0xf61b7744 in tokio::runtime::task::harness::poll_future<futures_util::future::future::Map<futures_util::future::try_future::MapErr<hyper::client::conn::Connection<tokio::net::tcp::stream::TcpStream, hyper::body::body::Body>, hyper::client::client::{impl#3}::connect_to::{closure#0}::{closure#0}::{async_block#0}::{closure_env#0}<hyper::client::connect::http::HttpConnector<hyper::client::connect::dns::GaiResolver>, hyper::body::body::Body>>, hyper::client::client::{impl#3}::connect_to::{closure#0}::{closure#0}::{async_block#0}::{closure_env#1}<hyper::client::connect::http::HttpConnector<hyper::client::connect::dns::GaiResolver>, hyper::body::body::Body>>, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle, alloc::alloc::Global>> (core=0xf5b071d8, cx=...)
    at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/harness.rs:488
#28 0xf61b8aa4 in tokio::runtime::task::harness::Harness<futures_util::future::future::Map<futures_util::future::try_future::MapErr<hyper::client::conn::Connection<tokio::net::tcp::stream::TcpStream, hyper::body::body::Body>, hyper::client::client::{impl#3}::connect_to::{closure#0}::{closure#0}::{async_block#0}::{closure_env#0}<hyper::client::connect::http::HttpConnector<hyper::client::connect::dns::GaiResolver>, hyper::body::body::Body>>, hyper::client::client::{impl#3}::connect_to::{closure#0}::{closure#0}::{async_block#0}::{closure_env#1}<hyper::client::connect::http::HttpConnector<hyper::client::connect::dns::GaiResolver>, hyper::body::body::Body>>, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle, alloc::alloc::Global>>::poll_inner<futures_util::future::future::Map<futures_util::future::try_future::MapErr<hyper::client::conn::Connection<tokio::net::tcp::stream::TcpStream, hyper::body::body::Body>, hyper::client::client::{impl#3}::connect_to::{closure#0}::{closure#0}::{asy--Type <RET> for more, q to quit, c to continue without paging--
nc_block#0}::{closure_env#0}<hyper::client::connect::http::HttpConnector<hyper::client::connect::dns::GaiResolver>, hyper::body::body::Body>>, hyper::client::client::{impl#3}::connect_to::{closure#0}::{closure#0}::{async_block#0}::{closure_env#1}<hyper::client::connect::http::HttpConnector<hyper::client::connect::dns::GaiResolver>, hyper::body::body::Body>>, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle, alloc::alloc::Global>> (self=<optimized out>) at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/harness.rs:209
#29 0xf61bb904 in tokio::runtime::task::harness::Harness<futures_util::future::future::Map<futures_util::future::try_future::MapErr<hyper::client::conn::Connection<tokio::net::tcp::stream::TcpStream, hyper::body::body::Body>, hyper::client::client::{impl#3}::connect_to::{closure#0}::{closure#0}::{async_block#0}::{closure_env#0}<hyper::client::connect::http::HttpConnector<hyper::client::connect::dns::GaiResolver>, hyper::body::body::Body>>, hyper::client::client::{impl#3}::connect_to::{closure#0}::{closure#0}::{async_block#0}::{closure_env#1}<hyper::client::connect::http::HttpConnector<hyper::client::connect::dns::GaiResolver>, hyper::body::body::Body>>, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle, alloc::alloc::Global>>::poll<futures_util::future::future::Map<futures_util::future::try_future::MapErr<hyper::client::conn::Connection<tokio::net::tcp::stream::TcpStream, hyper::body::body::Body>, hyper::client::client::{impl#3}::connect_to::{closure#0}::{closure#0}::{async_block#0}::{closure_env#0}<hyper::client::connect::http::HttpConnector<hyper::client::connect::dns::GaiResolver>, hyper::body::body::Body>>, hyper::client::client::{impl#3}::connect_to::{closure#0}::{closure#0}::{async_block#0}::{closure_env#1}<hyper::client::connect::http::HttpConnector<hyper::client::connect::dns::GaiResolver>, hyper::body::body::Body>>, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle, alloc::alloc::Global>> (self=...) at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/harness.rs:154
#30 0xf632a578 in tokio::runtime::scheduler::multi_thread::worker::{impl#1}::run_task::{closure#0} () at src/runtime/scheduler/multi_thread/worker.rs:598
#31 tokio::runtime::coop::with_budget<core::result::Result<alloc::boxed::Box<tokio::runtime::scheduler::multi_thread::worker::Core, alloc::alloc::Global>, ()>, tokio::runtime::scheduler::multi_thread::worker::{impl#1}::run_task::{closure_env#0}> (budget=..., f=...) at src/runtime/coop.rs:107
#32 tokio::runtime::coop::budget<core::result::Result<alloc::boxed::Box<tokio::runtime::scheduler::multi_thread::worker::Core, alloc::alloc::Global>, ()>, tokio::runtime::scheduler::multi_thread::worker::{impl#1}::run_task::{closure_env#0}> (f=...) at src/runtime/coop.rs:73
#33 tokio::runtime::scheduler::multi_thread::worker::Context::run_task (self=0xf52fdbac, task=..., core=0xf5b059d8) at src/runtime/scheduler/multi_thread/worker.rs:597
#34 0xf6329448 in tokio::runtime::scheduler::multi_thread::worker::Context::run (self=<optimized out>, core=0xf5b059d8) at src/runtime/scheduler/multi_thread/worker.rs:548
#35 0xf63192cc in tokio::runtime::scheduler::multi_thread::worker::run::{closure#0}::{closure#0} () at src/runtime/scheduler/multi_thread/worker.rs:513
#36 tokio::runtime::context::scoped::Scoped<tokio::runtime::scheduler::Context>::set<tokio::runtime::scheduler::Context, tokio::runtime::scheduler::multi_thread::worker::run::{closure#0}::{closure_env#0}, ()> (self=0xf4500618, t=<optimized out>, f=...) at src/runtime/context/scoped.rs:40
#37 0xf63186ac in tokio::runtime::context::set_scheduler::{closure#0}<(), tokio::runtime::scheduler::multi_thread::worker::run::{closure#0}::{closure_env#0}> (c=0xf45005f8) at src/runtime/context.rs:180
#38 std::thread::local::LocalKey<tokio::runtime::context::Context>::try_with<tokio::runtime::context::Context, tokio::runtime::context::set_scheduler::{closure_env#0}<(), tokio::runtime::scheduler::multi_thread::worker::run::{closure#0}::{closure_env#0}>, ()> (self=<optimized out>, f=...) at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/thread/local.rs:283
#39 std::thread::local::LocalKey<tokio::runtime::context::Context>::with<tokio::runtime::context::Context, tokio::runtime::context::set_scheduler::{closure_env#0}<(), tokio::runtime::scheduler::multi_thread::worker::run::{closure#0}::{closure_env#0}>, ()> (self=<optimized out>, f=...) at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/thread/local.rs:260
#40 0xf631eb6c in tokio::runtime::context::set_scheduler<(), tokio::runtime::scheduler::multi_thread::worker::run::{closure#0}::{closure_env#0}> (v=<optimized out>, f=...) at src/runtime/context.rs:180
#41 0xf6319120 in tokio::runtime::scheduler::multi_thread::worker::run::{closure#0} () at src/runtime/scheduler/multi_thread/worker.rs:508
#42 tokio::runtime::context::runtime::enter_runtime<tokio::runtime::scheduler::multi_thread::worker::run::{closure_env#0}, ()> (handle=<optimized out>, allow_block_in_place=true, f=...)
    at src/runtime/context/runtime.rs:65
#43 0xf6328e54 in tokio::runtime::scheduler::multi_thread::worker::run (worker=...) at src/runtime/scheduler/multi_thread/worker.rs:500
#44 0xf630f83c in tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure#0} () at src/runtime/scheduler/multi_thread/worker.rs:466
#45 tokio::runtime::blocking::task::{impl#2}::poll<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}, ()> (self=..., _cx=<optimized out>)
    at src/runtime/blocking/task.rs:42
#46 0xf63146e8 in tokio::runtime::task::core::{impl#6}::poll::{closure#0}<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>, tokio::runtime::blocking::schedule::BlockingSchedule> (ptr=0xf5b05c48) at src/runtime/task/core.rs:331
#47 tokio::loom::std::unsafe_cell::UnsafeCell<tokio::runtime::task::core::Stage<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>>>::with_mut<tokio::runtime::task::core::Stage<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>>, core::task::poll::Poll<()>, tokio::runtime::task::core::{impl#6}::poll::{closure_env#0}<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>, tokio::runtime::blocking::schedule::BlockingSchedule>> (self=0xf5b05c48, f=...) at src/loom/std/unsafe_cell.rs:16
#48 tokio::runtime::task::core::Core<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>, tokio::runtime::blocking::schedule::BlockingSchedule>::poll<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>, tokio::runtime::blocking::schedule::BlockingSchedule> (self=0xf5b05c38, cx=...) at src/runtime/task/core.rs:320
#49 0xf631acd0 in tokio::runtime::task::harness::poll_future::{closure#0}<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>, tokio::runtime::blocking::schedule::BlockingSchedule> () at src/runtime/task/harness.rs:500
#50 core::panic::unwind_safe::{impl#23}::call_once<core::task::poll::Poll<()>, tokio::runtime::task::harness::poll_future::{closure_env#0}<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>, tokio::runtime::blocking::schedule::BlockingSchedule>> (self=...)
    at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/panic/unwind_safe.rs:272
#51 std::panicking::try::do_call<core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future::{closure_env#0}<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>, tokio::runtime::blocking::schedule::BlockingSchedule>>, core::task::poll::Poll<()>> (data=<optimized out>)
    at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs:557
#52 std::panicking::try<core::task::poll::Poll<()>, core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future::{closure_env#0}<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>, tokio::runtime::blocking::schedule::BlockingSchedule>>> (
    f=<error reading variable: Cannot access memory at address 0xc>) at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs:521
--Type <RET> for more, q to quit, c to continue without paging--
#53 std::panic::catch_unwind<core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future::{closure_env#0}<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>, tokio::runtime::blocking::schedule::BlockingSchedule>>, core::task::poll::Poll<()>> (f=...)
    at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panic.rs:350
#54 0xf630b1bc in tokio::runtime::task::harness::poll_future<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>, tokio::runtime::blocking::schedule::BlockingSchedule> (core=0xf5b05c38, cx=...) at src/runtime/task/harness.rs:488
#55 0xf630aa04 in tokio::runtime::task::harness::Harness<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>, tokio::runtime::blocking::schedule::BlockingSchedule>::poll_inner<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>, tokio::runtime::blocking::schedule::BlockingSchedule> (self=<optimized out>) at src/runtime/task/harness.rs:209
#56 0xf630a928 in tokio::runtime::task::harness::Harness<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>, tokio::runtime::blocking::schedule::BlockingSchedule>::poll<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>, tokio::runtime::blocking::schedule::BlockingSchedule> (self=...) at src/runtime/task/harness.rs:154
#57 0xf630f18c in tokio::runtime::task::raw::RawTask::poll (self=...) at src/runtime/task/raw.rs:201
#58 tokio::runtime::task::UnownedTask<tokio::runtime::blocking::schedule::BlockingSchedule>::run<tokio::runtime::blocking::schedule::BlockingSchedule> (self=...) at src/runtime/task/mod.rs:473
#59 tokio::runtime::blocking::pool::Task::run (self=...) at src/runtime/blocking/pool.rs:160
#60 tokio::runtime::blocking::pool::Inner::run (self=<optimized out>, worker_thread_id=0) at src/runtime/blocking/pool.rs:518
#61 0xf6314cf4 in tokio::runtime::blocking::pool::{impl#6}::spawn_thread::{closure#0} () at src/runtime/blocking/pool.rs:476
#62 std::sys::backtrace::__rust_begin_short_backtrace<tokio::runtime::blocking::pool::{impl#6}::spawn_thread::{closure_env#0}, ()> (f=...)
    at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/sys/backtrace.rs:152
#63 0xf63120b8 in std::thread::{impl#0}::spawn_unchecked_::{closure#1}::{closure#0}<tokio::runtime::blocking::pool::{impl#6}::spawn_thread::{closure_env#0}, ()> ()
    at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/thread/mod.rs:538
#64 core::panic::unwind_safe::{impl#23}::call_once<(), std::thread::{impl#0}::spawn_unchecked_::{closure#1}::{closure_env#0}<tokio::runtime::blocking::pool::{impl#6}::spawn_thread::{closure_env#0}, ()>>
    (self=<error reading variable: Cannot access memory at address 0x0>) at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/panic/unwind_safe.rs:272
#65 std::panicking::try::do_call<core::panic::unwind_safe::AssertUnwindSafe<std::thread::{impl#0}::spawn_unchecked_::{closure#1}::{closure_env#0}<tokio::runtime::blocking::pool::{impl#6}::spawn_thread::{closure_env#0}, ()>>, ()> (data=<optimized out>) at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs:557
#66 std::panicking::try<(), core::panic::unwind_safe::AssertUnwindSafe<std::thread::{impl#0}::spawn_unchecked_::{closure#1}::{closure_env#0}<tokio::runtime::blocking::pool::{impl#6}::spawn_thread::{closure_env#0}, ()>>> (f=<error reading variable: Cannot access memory at address 0x0>) at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs:521
#67 std::panic::catch_unwind<core::panic::unwind_safe::AssertUnwindSafe<std::thread::{impl#0}::spawn_unchecked_::{closure#1}::{closure_env#0}<tokio::runtime::blocking::pool::{impl#6}::spawn_thread::{closure_env#0}, ()>>, ()> (f=<error reading variable: Cannot access memory at address 0x0>) at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panic.rs:350
#68 std::thread::{impl#0}::spawn_unchecked_::{closure#1}<tokio::runtime::blocking::pool::{impl#6}::spawn_thread::{closure_env#0}, ()> ()
    at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/thread/mod.rs:537
#69 core::ops::function::FnOnce::call_once<std::thread::{impl#0}::spawn_unchecked_::{closure_env#1}<tokio::runtime::blocking::pool::{impl#6}::spawn_thread::{closure_env#0}, ()>, ()> ()
    at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/ops/function.rs:250
#70 0xf64da77c in alloc::boxed::{impl#48}::call_once<(), dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global> () at library/alloc/src/boxed.rs:2070
#71 alloc::boxed::{impl#48}::call_once<(), alloc::boxed::Box<dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global>, alloc::alloc::Global> () at library/alloc/src/boxed.rs:2070
#72 std::sys::pal::unix::thread::{impl#2}::new::thread_start () at library/std/src/sys/pal/unix/thread.rs:108
#73 0xf757b99e in start_thread (arg=0xe2bfcba) at pthread_create.c:477
#74 0xf751402c in ?? () at ../sysdeps/unix/sysv/linux/arm/clone.S:73 from /lib/arm-linux-gnueabihf/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb)
kingosticks commented 1 month ago

Do you have the same versions of Gstreamer in Docker and non-Docker armhf setups? Can you try the latest release at https://github.com/kingosticks/gst-plugins-rs-build/releases/tag/gst-plugin-spotify_0.14.0-alpha.1-1 ? I don't think there's anything we can do about this here other than try and work out which upstream to take this to.

mczerski commented 1 month ago

Yes, versions are the same. also tried v0.14.0 - same result. Actually on the other host spotify works also in docker so I suspect that this issue is somehow related to the machine rather than docker itself. Unfortunetly I cannot test without docker on this machine (CoreElec)