Shadowsocks crashing and exiting on startup

[lavaguy1]

This is a new problem for me - since yesterday 24.7.20, I'm getting this in my PIA container log:

2020-07-24T11:03:31.113+0200    ERROR   shadowsocks: shadowsocks error: Error relocating /usr/lib/libcork.so.15: __gettimeofday_time64: symbol not found
2020-07-24T11:03:31.114+0200    ERROR   shadowsocks: shadowsocks error: Error relocating /usr/lib/libcork.so.15: __stat_time64: symbol not found
2020-07-24T11:03:31.114+0200    ERROR   shadowsocks: shadowsocks error: Error relocating /usr/lib/libcork.so.15: __localtime64_r: symbol not found
2020-07-24T11:03:31.115+0200    ERROR   shadowsocks: shadowsocks error: Error relocating /usr/lib/libcork.so.15: __gmtime64_r: symbol not found
2020-07-24T11:03:31.116+0200    ERROR   shadowsocks: shadowsocks error: Error relocating /usr/bin/ss-server: __time64: symbol not found
2020-07-24T11:03:31.117+0200    ERROR   shadowsocks: shadowsocks error: Error relocating /usr/bin/ss-server: __localtime64: symbol not found
2020-07-24T11:03:31.117+0200    ERROR   shadowsocks: exit status 127
2020-07-24T11:03:31.118+0200    INFO    shadowsocks: retrying in 1 minute

running on Raspbian, on Raspberry PI 4 (Linux pi4 5.4.51-v7l+ #1327 SMP Thu Jul 23 11:04:39 BST 2020 armv7l GNU/Linux)

Anyone have any tips?

Thanks in advance,

Steve

[qdm12]

Related to the latest commit. Using the standard alpine package repository failed to install shadowsocks (bad signature) for some reason. I changed it to the Netherlands mirror to fix it. Well it seems that fixed the installation but it installs a non working version... I'll try another repository. Thanks for reporting!

[frepke]

Hmmm, no Shadowsocks problems for me.

[qdm12]

Yes it should fail only on arm32v6/v7 and 386 I think. Arm64 and amd64 were working fine. What cpu arch are you running on again Frepke? Maybe it's a 32 bit/64 bit related issue. I'll look into compiling shadowsocks from source otherwise.

[frepke]

Running on an AMD64

[lavaguy1]

Hi, mine is running on Arm7 ... everything worked fine until picking up the latest container image... now no connection, dns tls doesn't seem to work and neither does ShadowSocks

[lavaguy1]

Super, thanks for the tip! I'll keep watching for the update.

[qdm12]

It should work now, it seems Alpine fixed their signature error for Shadowsocks. (wait for build to finish). Let me know and feel free to close the issue.

[lavaguy1]

Thanks, still need to try the latest version...

On Sat, Jul 25, 2020 at 5:21 PM Quentin McGaw notifications@github.com wrote:

It should work now, it seems Alpine fixed their signature error for Shadowsocks. (wait for build to finish https://github.com/qdm12/private-internet-access-docker/runs/909946999?check_suite_focus=true). Let me know and feel free to close the issue.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/qdm12/private-internet-access-docker/issues/211#issuecomment-663867190, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFHXQC5ZLIOMUENS2X25BRDR5LZ67ANCNFSM4PGQ3TOA .

[lavaguy1]

Sorry to bother about this again, but I am getting the same behavior again - shadowsocks fails with the exact same errors as reported above, when using v3.1.0 or latest... I am using 32bit Arm Raspbian on a Pi 4 b. The newest build that works is 3.0.1, and it seems to have issues with DOT...

[qdm12]

Ok I'll dig into it this weekend. For instance, on v3.0.1 what's the shadowsocks version logged at the top?

I might just try coding shadowsocks directly in the Go entrypoint program to remove that flaky dependency from Alpine.

[lavaguy1]

I'll check and let you know tomorrow... it's 3:30 on a work night here...

[lavaguy1]

the ShadowSocks version reported at start is 3.3.4

[qdm12]

Sorry I didn't get the time to implement shadowsocks from scratch over the weekend, will try the coming evenings.

On the other hand, I see Alpine Shadowsocks builds are from January so it shouldn't be a recent update that broke it 🤔

But from the logs, it's some C weird dependency problem. Maybe try with the Dockerfile:

FROM qmcgaw/private-internet-access
RUN apk add build-base

and build it with

docker build -t gluetun-test .

And replace qmcgaw/private-internet-access by gluetun-test in your run command. That might work although it makes the docker image like 300MB bigger. Maybe ARM requires some C libs for whatever Alpine related reason.

[lavaguy1]

I'll try that... I'm pretty new with docker, though - I haven't actually ever built a container with build before... ;)

Thanks for the quick reply!

[qdm12]

Let me know if you need help. Here are more detailed commands to run on your raspberry pi. Run these in a terminal:

mkdir -p ~/Desktop/gluetun
cd ~/Desktop/gluetun
echo "FROM qmcgaw/private-internet-access" > Dockerfile
echo "RUN apk add build-base" >> Dockerfile
docker build -t gluetun-test .
docker run -it --rm -e SHADOWSOCKS=on -e USER=blabla -e PASSWORD=blabla gluetun-test

And see if shadowsocks can start properly. Press CTRL+C to quit and remove the container.

[lavaguy1]

Awesome! thanks so much. I think I found my problem with this bit... Does it have to be in a project folder called "gluetun" (i used "~/docker" where my composefiles are)

my first attempt looked like this (i made a text file called dockerfile with the 'FROM' and 'RUN' lines in it):

user@machine:~/docker $ docker build -t gluetun-test . Sending build context to Docker daemon 16.14GB Step 1/2 : FROM gmcgaw/private-internet-access pull access denied for gmcgaw/private-internet-access, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

[lavaguy1]

ok. tried this:

user@machine:~/docker/gluetun/docker login -p blabla -u username
WARNING! Your password will be stored unencrypted in /home/user/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded
user@machine:~/docker/gluetun $ docker build -t gluetun-test .
Sending build context to Docker daemon  2.048kB
Step 1/2 : FROM gmcgaw/private-internet-access
pull access denied for gmcgaw/private-internet-access, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

no joy.

[qdm12]

Typo alert 🚒

It's qmcgaw not gmcgaw, then it should be able to pull even without having to login Docker. Docker thought it was a private image because of the typo.

[lavaguy1]

duh! thanks... I'm getting too old. it's always either my bad eyesight or sleep-deprivation...

On Mon, Aug 10, 2020 at 6:20 PM Quentin McGaw notifications@github.com wrote:

Typo alert 🚒

It's qmcgaw not gmcgaw, then it should be able to pull even without having to login Docker. Docker thought it was a private image because of the typo.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/qdm12/gluetun/issues/211#issuecomment-671452735, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFHXQC5GTWC3SDATKJUP5K3SAAM5TANCNFSM4PGQ3TOA .

[lavaguy1]

hmmm... that builds now, but it looks like ShadowSocks is dropping an error...

[qdm12]

Bummer 😕

Run Alpine interactively (from anywhere)

docker run -it --rm alpine:3.12

Enter:

echo "http://nl.alpinelinux.org/alpine/edge/testing" >> /etc/apk/repositories
apk add --update shadowsocks-libev
ss-server
echo $?
exit

Does ss-server work? What's the output exit code from $??

I have a raspberry pi 3b (somewhere hiding), I'll dig it out and try on my end. But hey at least now you know how to build docker images!

[lavaguy1]

Thanks for the detailed help, and yes I have learned a bit!

But, I did that, and got this result:

[qdm12]

retry these steps but install as well

apk add --update libcork-dev

before calling ss-server and let's see if any error shows.

I'm in the process of setting up my Raspberry Pi with Docker etc. but my god it's painfully slow, especially when coming from a 6 cores ryzen cpu 😄 I'll check about the raspberry pi 4, it might be decently faster.

[lavaguy1]

thanks i'll try. an overclocked pi4 is actually pretty snappy.

On Tue, Aug 11, 2020 at 2:20 AM Quentin McGaw notifications@github.com wrote:

retry these steps but install as well

apk add --update libcork-dev

before calling ss-server and let's see if any error shows.

I'm in the process of setting up my Raspberry Pi with Docker etc. but my god it's painfully slow, especially when coming from a 6 cores ryzen cpu 😄 I'll check about the raspberry pi 4, it might be decently faster.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/qdm12/gluetun/issues/211#issuecomment-671653825, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFHXQC4UO2GYFLRPL53AI7DSACFFFANCNFSM4PGQ3TOA .

[lavaguy1]

sorry, that did the same error list with symbol not found:

Error relocating /usr/lib/libcork.so.15: __gettimeofday_time64: symbol not found
Error relocating /usr/lib/libcork.so.15: __stat_time64: symbol not found
Error relocating /usr/lib/libcork.so.15: __localtime64_r: symbol not found
Error relocating /usr/lib/libcork.so.15: __gmtime64_r: symbol not found
Error relocating /usr/bin/ss-server: __time64: symbol not found
Error relocating /usr/bin/ss-server: __localtime64: symbol not found

[qdm12]

I tried putting it every way on my raspberry pi, it won't work either 😢 They wrong-compiled their ss-server binary for ARM it seems and Alpine doesn't keep older versions of shadowsocks in their edge/testing category.

I wanted to compile it myself but the instructions seems overkill and I don't want to bloat the Dockerfile even more.

Anyway it has been long I wanted to do it with a Go program, let me try to do a quick fix with go-shadowsocks2 tomorrow.

For instance what you could do - a bit stretched but it should work - is to run a connected Shadowsocks container to gluetun, for example have:

Gluetun with Shadowsocks off but the port published

docker run ... -p 8388:8388/udp -p 8388:8388/tcp -e SHADOWSOCKS=off gluetun

And the connected Shadowsocks container

docker run ... -e PASSWORD=<password> --network_mode="container:gluetun" shadowsocks/shadowsocks-libev

And then you can access Shadowsocks as before (untested & commands are not exact). Let me know I can write you a docker-compose.yml file to run these two containers if you need.

A bit convoluted, but it should work 😄

[lavaguy1]

Thanks a lot for your generous offer! Please take your time.

I'll give the secondary container approach a try (I'm actually doing that on an older machine that I'm replacing with this new one, but using the container "serjs/go-socks5-proxy" which pretty much works. Not sure if it's optimized, and it certainly isn't as feature-rich as ShadowSocks seems to be.

On Tue, Aug 11, 2020 at 4:26 AM Quentin McGaw notifications@github.com wrote:

I tried putting it every way on my raspberry pi, it won't work either 😢 They wrong-compiled their ss-server binary for ARM it seems and Alpine doesn't keep older versions of shadowsocks in their edge/testing category.

I wanted to compile it myself but the instructions https://github.com/shadowsocks/shadowsocks-libev#linux seems overkill and I don't want to bloat the Dockerfile even more.

Anyway it has been long I wanted to do it with a Go program, let me try to do a quick fix with go-shadowsocks2 https://github.com/shadowsocks/go-shadowsocks2 tomorrow.

For instance what you could do - a bit stretched but it should work - is to run a connected Shadowsocks container to gluetun, for example have:

Gluetun with Shadowsocks off but the port published

docker run ... -p 8388:8388/udp -p 8388:8388/tcp -e SHADOWSOCKS=off gluetun

And the connected Shadowsocks container

docker run ... -e PASSWORD= --network_mode="container:gluetun" shadowsocks/shadowsocks-libev

And then you can access Shadowsocks as before (untested & commands are not exact). Let me know I can write you a docker-compose.yml file to run these two containers if you need.

A bit convoluted, but it should work 😄

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/qdm12/gluetun/issues/211#issuecomment-671686667, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFHXQC5WGRJCXJCVPK3PDMDSACT4BANCNFSM4PGQ3TOA .

[lavaguy1]

Also, is there any chance you could open a ticket on Alpine that doesn't sound like a monkey trying to describe the low gravity Space Station toilet? I'm not sure if I can...

On Tue, Aug 11, 2020 at 4:26 AM Quentin McGaw notifications@github.com wrote:

I tried putting it every way on my raspberry pi, it won't work either 😢 They wrong-compiled their ss-server binary for ARM it seems and Alpine doesn't keep older versions of shadowsocks in their edge/testing category.

I wanted to compile it myself but the instructions https://github.com/shadowsocks/shadowsocks-libev#linux seems overkill and I don't want to bloat the Dockerfile even more.

Anyway it has been long I wanted to do it with a Go program, let me try to do a quick fix with go-shadowsocks2 https://github.com/shadowsocks/go-shadowsocks2 tomorrow.

For instance what you could do - a bit stretched but it should work - is to run a connected Shadowsocks container to gluetun, for example have:

Gluetun with Shadowsocks off but the port published

docker run ... -p 8388:8388/udp -p 8388:8388/tcp -e SHADOWSOCKS=off gluetun

And the connected Shadowsocks container

docker run ... -e PASSWORD= --network_mode="container:gluetun" shadowsocks/shadowsocks-libev

And then you can access Shadowsocks as before (untested & commands are not exact). Let me know I can write you a docker-compose.yml file to run these two containers if you need.

A bit convoluted, but it should work 😄

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/qdm12/gluetun/issues/211#issuecomment-671686667, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFHXQC5WGRJCXJCVPK3PDMDSACT4BANCNFSM4PGQ3TOA .

[qdm12]

Done, I flagged the shadowsocks-libev package. I'm working on integrating Shadowsocks directly in the Go code but it might take more than one day. I'll ping back here once it's done.

[lavaguy1]

thanks again! no pressure, whenever you've got time.

[qdm12]

Quick update, I'm like 80% done, I made another whole Docker image to run a Shadowsocks server (going a bit off the goal here but why not)... It's working well, I'm just polishing off the code to have as a nice Go API so that I can easily plug it in gluetun. Should be done this weekend 👍

[qdm12]

Alright my Go Shadowsocks server is ready, I just need to plug it in Gluetun now. Will do it tomorrow 😉

[lavaguy1]

Thanks so much Quentin! looking forward to taking it for a spin when it's done!

On Sun, Aug 16, 2020 at 12:34 AM Quentin McGaw notifications@github.com wrote:

Alright my Go Shadowsocks server is ready https://github.com/qdm12/ss-server, I just need to plug it in Gluetun now. Will do it tomorrow 😉

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/qdm12/gluetun/issues/211#issuecomment-674453711, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFHXQC7REKTPMARNVFYW2M3SA4EQPANCNFSM4PGQ3TOA .

[qdm12]

Can you please try (no hurry) with the image qmcgaw/private-internet-access:shadowsocks? It seems to work for TCP for me, but haven't tested UDP so far. You might also want to set -e SHADOWSOCKS_LOG=on to get some information.

[lavaguy1]

that was fast! I'll try it tomorrow night, my time (CET).

Sorry if it's getting old, but thanks for working on this!

On Mon, Aug 17, 2020 at 1:32 AM Quentin McGaw notifications@github.com wrote:

Can you please try (no hurry) with the image qmcgaw/private-internet-access:shadowsocks? It seems to work for TCP for me, but haven't tested UDP so far. You might also want to set -e SHADOWSOCKS_LOG=on to get some information.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/qdm12/gluetun/issues/211#issuecomment-674592450, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFHXQC7DQ3775ZVP7JLDCGLSBBUARANCNFSM4PGQ3TOA .

[lavaguy1]

hmmm... that seems to have gone badly...

I added the new image name to my docker-compose.yml and did the whole:

pi@pi4:~/docker $ docker-compose stop && docker-compose rm -f ... pi@pi4:~/docker $ docker-compose up --force-recreate --always-recreate-deps --remove-orphans -d Creating pia ... done Creating portainer ... done Creating airsonic ... done Creating heimdall ... error

and then it dumped a lot of errors about pia container not being started yet (i have all my server processes routing over the pia container with network directives in my compose.yml)

in the pia container itself i see:

2020-08-17T21:54:31.463209691Z [FATAL tini (6)] exec /entrypoint failed: Exec format error

2020-08-17T21:54:37.332465839Z [FATAL tini (6)] exec /entrypoint failed: Exec format error

2020-08-17T21:54:42.643346627Z [FATAL tini (6)] exec /entrypoint failed: Exec format error

2020-08-17T21:54:47.635999278Z [FATAL tini (6)] exec /entrypoint failed: Exec format error

2020-08-17T21:54:52.630045564Z [FATAL tini (7)] exec /entrypoint failed: Exec format error

2020-08-17T21:54:57.450812121Z [FATAL tini (6)] exec /entrypoint failed: Exec format error

2020-08-17T21:55:03.489404244Z [FATAL tini (7)] exec /entrypoint failed: Exec format error

2020-08-17T21:55:13.429993775Z [FATAL tini (7)] exec /entrypoint failed: Exec format error

2020-08-17T21:55:29.106910084Z [FATAL tini (7)] exec /entrypoint failed: Exec format error

2020-08-17T21:55:57.699688877Z [FATAL tini (6)] exec /entrypoint failed: Exec format error

On Mon, Aug 17, 2020 at 1:32 AM Quentin McGaw notifications@github.com wrote:

Can you please try (no hurry) with the image qmcgaw/private-internet-access:shadowsocks? It seems to work for TCP for me, but haven't tested UDP so far. You might also want to set -e SHADOWSOCKS_LOG=on to get some information.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/qdm12/gluetun/issues/211#issuecomment-674592450, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFHXQC7DQ3775ZVP7JLDCGLSBBUARANCNFSM4PGQ3TOA .

[qdm12]

~Hm strange, are you sure you are using the :shadowsocks tag?~

~Try with~

docker run -it --rm --cap-add=NET_ADMIN -e REGION="Switzerland" -e USER=js89ds7 -e PASSWORD=8fd9s239G -e SHADOWSOCKS=on -e SHADOWSOCKS_LOG=on qmcgaw/private-internet-access:shadowsocks

~?~

EDIT: Oh never mind, the :shadowsocks tag is only built for amd64, let me fix that.

[qdm12]

As mentioned in my edit, sorry that image was not built for arm and other architectures.

I made a release v3.2.0-rc1 on Github, that should had triggered a cross-architecture build with the Docker tag :v3.2.0-rc1.

Please try with the tag :v3.2.0-rc1 instead.

[frepke]

With the tag :v3.2.0-rc1 sometimes the cpu load is exceptional high. In HTOP I read CPU-loads of 380% (/entrypoint). I think it's 100% per core (4-core machine).

[lavaguy1]

will do! thanks, sorry for forgetting to mention that I'm on arm 32bit.

[lavaguy1]

Ok, that one worked fine - pulled and started ok, works fine withvShadowSocks for windows (v. 4.1.10.0) as a local proxy.

[lavaguy1]

whoops. Yeah, i can verify... htop showing extremely high CPU utilization up to 230% for me.

[qdm12]

Oops, it seems like I rushed Shadowsocks a bit too much. Give me a few more days 😄

[lavaguy1]

no problem - cpu load seems to drop down a lot after running for a while... I've been running since my last post. When ShadowSocks isn't actually processing web requests, it seems to be well behaved. It does spike cpu when it is processing, though, the individual "/entrypoint" processes all go up to about 15-25% cpu (according to HTOP) and it takes a little while to settle down (ads, maybe?). I'll keep it running until it gets in the way of something and see how it goes...

[qdm12]

I'll profile resources for the standalone shadowsocks program I wrote to see where the crazy cpu spikes come from.

Also note that now shadowsocks is part of /entrypoint (well I should change the name to gluetun haha). The Go version might also be a tiny bit more cpu taxing than the C version although it shouldn't be such a big change I think. Maybe try with another encryption method (SHADOWSOCKS_METHOD=aes_128_gcm).

[frepke]

I'll profile resources for the standalone shadowsocks program I wrote to see where the crazy cpu spikes come from.

Also note that now shadowsocks is part of /entrypoint (well I should change the name to gluetun haha). The Go version might also be a tiny bit more cpu taxing than the C version although it shouldn't be such a big change I think. Maybe try with another encryption method (SHADOWSOCKS_METHOD=aes_128_gcm).

Those aren't only spikes, some periods are more than 15 minutes long.

[qdm12]

Alright so it seems cpu usage steadily increases, I'm probably forgetting to close connections/threads somewhere. It also reaches 100% on my 8 cores after a few web page loads 😄 So yes definitely a problem! I'm still investigating. Sorry if that messed up your device/services when trying it.

[lavaguy1]

no problem here... easy to switch back.

[qdm12]

Should be fixed 👍 It was essentially changing a continue to a return n, nil 😄 But that was challenging to find haha

Try with Docker image tag :3.2.0-rc2

[frepke]

Should be fixed 👍 It was essentially changing a continue to a return n, nil 😄 But that was challenging to find haha

Try with Docker image tag :3.2.0-rc2

Running for 2 hours now without problems or high cpu-usage. I noticed a little modification in the Shadowsocks INFO, /TCP in the info is missing 😄