Kabe0 / deluge-windscribe

To provide an isolated VPN layer with Deluge
23 stars 6 forks source link

"Unable to use default config directory, exiting..." #5

Closed Hukuma1 closed 4 years ago

Hukuma1 commented 4 years ago

Running the container via Portainer and this is what pops up in the logs. I left the volume /config/ folder mount at default once it pulls the image.

[ERROR ][deluge.common:131 ] Unable to use default config directory, exiting... ([Errno 13] Permission denied: '/config/.config')

Workaround in mean time is to bind my own folder with full permissions.

Hukuma1 commented 4 years ago

Although under Deluge WebUI preferences, it's telling me downloads will go to /config/Downloads folder. Even though my /downloads mount is set to different download folder?

Kabe0 commented 4 years ago

Hi @Hukuma1

Just to clarify, instead of binding a directory your using a docker volume? The default UID and GID of the container is 1000:1000 as is set with DEL_UID and DEL_GID. That might be the issue? I will take a look at the volume binding on my local to see if I can reproduce the problem.

The downloads folder was an oversight on my part. It's defaulting to the home directory which is set to the /config folder. I will work on a fix for this some time this week. You can set the path to /downloads in the deluge config settings. It will save the changes.

Hukuma1 commented 4 years ago

I'm pretty new to Docker/Portainer world so please bear with me. And thank you for your help/understanding.

Let's first make sure I installed it correctly with the proper settings passed on.

wind1

Port 8112 is the only one exposed, is this right?

wind2

For the /config folder on default pull/run, the container was set to a volume of some sort. I ended up making a /config in /mnt/dietpi_userdata and gave it full permissions. And switched the default volume, to this bind you see. Seems to be seeing it/working.

wind3

I've seen mentions of adding DNS servers for Windscribe, not sure if this is correct? Added Google DNS servers currently.

wind4

I originally was using VPN_USERNAME/VPN_PASSWORD/VPN_LOCATION, but noticed it still defaulted to trying to find/use/force VPN_AUTH (even if I removed VPN_AUTH from the envs, it would re-create it). Possible bug? So I simply made auth.conf file and placed it into my config folder with my credentials. No biggie.

wind5 Added the device.

wind6 Enabled NET_ADMIN.

From this setup here is my log from the container:

Starting windscribe ... OK,
usermod: no changes,

Loading up Deluge webUI on port 8112 I see this: (after inputting default deluge password)

wind7 I hit connect and it connects successfully.

wind8 I then decide to test it and add a torrent file. I get an error message, nothing will download.

wind9 I terminal in and run windscribe status

wind10 I then run windscribe connect and receive this. It does connect, but now I'm unable to access the Deluge WebUI.

I check my Windscribe stats and it seems to be using my data allowance. So it's downloading! I just can't access Deluge to see it. Once I issue windscribe disconnect I can immediately access Deluge WebUI and see my download progress. It also stalls out as I have no more connection to Windscribe. So I guess that's good confirmation that nothing will download if I lose Windscribe connection.

wind11

My Deluge WebUI preferences are completely default (minus the /config/Downloads folder change I made as per your suggestion).

So the only questions are: how do I get it to connect automatically to Windscribe on container start, and how do I keep Deluge WebUI not being blocked while downloading? And also, is this the most secure method of using your container? Heard about some DNS leaking with UPnP or something? (Windscribe specific possibly)

Kabe0 commented 4 years ago

Environment variables will always set, as that's the default behaviour for Portainer. If you specify the VPN_USERNAME VPN_PASSWORD it will no longer try to load the config file even while being set. If you find setting the variables without the file prevents Windscribe from logging in, let me know.

For the second part, I see your using raspberry-pi.

Yea, something is wrong with Docker binding for networks on pi I have noticed. It's interesting that Windscribe is blocking the webui, as what is happening is the firewall is turned on for Windscribe, which protects you from having your IP exposed, but it's quite possible that it's also blocking the local user 127.0.0.1 which is needed for binding. I might have to figure out a way to expose the iptables and specify the 127.0.0.1 explicitly to get the view to work correctly...

The frustrating thing is that this issue does not happen on normal x86/x64 processors. So it appears to be a pi specific docker issue. I can take a look this weekend as I won't have access to a pi computer till then.

What pi version are you using?

Hukuma1 commented 4 years ago

Ah I see. Yeah, looks like not having the auth.conf file anywhere (even with VPN_AUTH=/config/auth.conf env) but passing the VPN_USERNAME/VPN_PASSWORD does allow me to login manually by issuing windscribe connect. It does not seem to use my provided VPN_LOCATION though? Probably using BEST flag and not my custom one. And just to confirm, is your container supposed to auto-login on start of the container to windscribe? Or I'm supposed to do it manually?

As for the second part, yeah RPi4. I hope we can figure it out because otherwise this is a great plug-n-play solution! Let me know if I can help in solving this issue for anyone else on a Pi.

I came across this, but wasn't sure how to incorporate or if it's even the root of the issue: https://www.reddit.com/r/Windscribe/comments/cn2qfb/etcresolvconf_is_not_a_symlink_this_may_break_dns/

pearlythepirate commented 4 years ago

I seem to be getting this same behavior on my x64 Docker install on Linux as well. Disconnecting Windscribe using the command line will allow access to the WebUI but not while Windscribe is connected.

Kabe0 commented 4 years ago

@Hukuma1 It's suppose to automatically connect to the windscribe server on launch. If it fails to connect it will kill the docker container rather than continuing. Literally the code is, get username and password from environment variables, if it's not found, load the auth file. Either way, it's suppose to start windscribe. You have the same Pi version I do which docker does not 100% support yet. The resolve.conf should not be an issue, but it's worth looking into.

When you see the logs for booting up does it look like the following? Initializing Container Starting windscribe ... OK Initializing Deluge usermod: no changes Deluged Init

@pearlythepirate That is really odd. I am a bit baffled that this has never been a problem for me on my main os Ubuntu. I wonder if it's distro dependent. What distro are you running docker on? Can you test something for me. Try only disabling the firewall instead of the whole windscribe service.... run windscribe firewall off and see if the webui starts to appear.

I will try to reproduce the problem this weekend. If I can get the issue on my main system, then it will be a lot easier to track down the problem.

pearlythepirate commented 4 years ago

@pearlythepirate What distro are you running docker on? Can you test something for me. Try only disabling the firewall instead of the whole windscribe service.... run windscribe firewall off and see if the webui starts to appear.

Ubuntu 19.10 (Server) windscribe firewall off does not allow the webui to appear.

Kabe0 commented 4 years ago

Ok, that distro should be working, however the latest I tested was 18.04...

When the container is running, try executing the following to see if the port is open sudo lsof -i -P -n | grep 8112

You should see the following docker-pr 26905 root 4u IPv6 197727 0t0 TCP *:8112 (LISTEN

As well try connecting on the server using curl http://127.0.0.1:8112. If it has HTML, that means the webui is running at least locally. Are you running the ufw firewall?

pearlythepirate commented 4 years ago
matt@the-source:~$ sudo ufw status
Status: inactive
matt@the-source:~$ curl http://127.0.0.1:8112
curl: (7) Failed to connect to 127.0.0.1 port 8112: Connection refused
matt@the-source:~$ sudo lsof -i -P -n | grep 8112
matt@the-source:~$
matt@the-source:~$ sudo lsof -i -P -n | grep 9008
docker-pr  5802            root    4u  IPv6 1004302      0t0  TCP *:9008 (LISTEN)
matt@the-source:~$ curl http://127.0.0.1:9008
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
    <head>
        <title>Deluge WebUI 2.0.3-2-201906121747-ubuntu18.04.1</title>

The commands both seem to work on port 9008 which was what I have docker forwarding port 8112 to.

Hukuma1 commented 4 years ago

@Hukuma1 It's suppose to automatically connect to the windscribe server on launch. If it fails to connect it will kill the docker container rather than continuing. Literally the code is, get username and password from environment variables, if it's not found, load the auth file. Either way, it's suppose to start windscribe. You have the same Pi version I do which docker does not 100% support yet. The resolve.conf should not be an issue, but it's worth looking into.

When you see the logs for booting up does it look like the following? Initializing Container Starting windscribe ... OK Initializing Deluge usermod: no changes Deluged Init

Maybe that's the issue.

All I see is this in logs:

Starting windscribe ... OK,
usermod: no changes,

It does not stop the container (if it did I would not be able to console in using Portainer to issue connect command manually).

Also in Deluge I see this in the preferences: deluge

Nothing is "enabled." I also have to connect manually to the daemon. So seems like container is not doing any of these things automatically on my setup.

Same thing with curl http://127.0.0.1:8112 as @pearlythepirate, I can connect and see output of the WebUI html just fine.

Edit: I'm checking the logs live in Portainer. Is there an internal log inside of the container that maybe is more verbose?

Kabe0 commented 4 years ago

@Hukuma1 The webui is not going to be enabled as we are using the deluge-web daemon for the view client. webui plugin is only used when you have deluge setup without the daemon. It may still be running the webui but it's possible that windscribe is outputting something that the python script is not liking as it checks the return result of windscribe to make sure no weird messages happen. I will have to see for sure...

@Hukuma1 @pearlythepirate Ok so if you guy's are getting a curl output, that means that both containers would be visible on the actual client that is running the container. So the deluge-webui is running, but it's not accessible outside the local system running the container.

The issue appears to be isolated to the fact that the port that docker opens 8112 or 9008 are not actually forwarding outside the local environment due to windscribe modifying the iptables. It's like the interface loopback is not working from your local network to dockers own bridge network. I had the same problem on my own Pi4 so I will take a stab at fixing the issue.

Kabe0 commented 4 years ago

@Hukuma1 to see more verbose logs, the only option right now is to look at the /var/log folder which contains logs for deluge and system messages.

Hukuma1 commented 4 years ago

I just checked and they're mostly blank.

At this point this issue is beyond my scope of knowledge, but then I found this as well.

https://dev.deluge-torrent.org/wiki/UserGuide/ThinClient#WebUI Is the section underneath talking about connecting to IP 127.0.0.2 of any relevance?

Either way, appreciate it if you take a look this weekend when you can. Thanks!

Hukuma1 commented 4 years ago

Any progress on this? :(

Kabe0 commented 4 years ago

I may have something, I just need to test with two computers... I can commit it if you want to build and test it, im just not ready to publish it yet... It's on https://github.com/Kabe0/deluge-windscribe/tree/dev-firewall-1

I am first trying to add to the IPTables, but if that does not work, i'm going to have to just manually configure the firewall instead of having Windscribe do it.

Sorry for the delay, have been doing a bunch of wedding plan stuff which is consuming my weekends so i'm not getting as much time as I want to tinker with this problem :(

I think I am on the right track though, so I think I will have something maybe in a few days if the Windscribe service doesn't do something to mess me up.

Hukuma1 commented 4 years ago

Any chance you could push it to docker hub as a test branch? I'm not certain how to pull the updates off Github into the container.

And congrats on the wedding!! :)

Kabe0 commented 4 years ago

Sure I'll push it tomorrow @Hukuma1

Kabe0 commented 4 years ago

Ok it's been pushed on kabe0/deluge-windscribe:dev... I still need to personally test this myself. Had to go into a bunch of meeting this weekend due to the whole virus fiasco.

Hukuma1 commented 4 years ago

Using this file: https://hub.docker.com/layers/kabe0/deluge-windscribe/dev-armv7/images/sha256-7003dc08500eaa03976ebd9e2d0fc9404700a7d99e92645d6017c0ddfb289463?context=explore

Thanks for doing that. Just tried. It seems to be working exactly the same way as your stable branch. I still do not have it autologin and once I manually connect it the deluge-web is inaccessible. :(

Starting windscribe ... OK,
usermod: no changes,
Kabe0 commented 4 years ago

Long time no update. Sorry about that! Finally found some time to actually take a look.

Pushed in an update 1.7.0. Please try it out. Works for me on my RaspberriPi 4.

The first time I ran it it did not work as intended, so I had to restart my Pi before it would function properly. It appears that the Pi IP tables are a bit more touchy than on Ubuntu.

Kabe0 commented 4 years ago

Also added in an adjustment that will force the deluge to create a first time config so the /downloads folder will be set by default on a first run if no config folder exists.

Please let me know if any issues exists for the VPN layer. It should now set the outgoing interface to tun0 automatically.

@Hukuma1

Hukuma1 commented 4 years ago

Thanks for the update!

Here is what I'm getting with the new build:

Starting windscribe ... OK,
usermod: no changes,
21:48:23 [ERROR   ][deluge.common:131 ] Unable to use default config directory, exiting... ([Errno 13] Permission denied: '/config/.config'),
21:48:23 [ERROR   ][deluge.common:131 ] Unable to use default config directory, exiting... ([Errno 13] Permission denied: '/config/.config'),
Configuring firewall settings,
Making config directory.,
Deluged Init,
Initializing Container,
Using VPN_USERNAME and VPN_PASSWORD to login.,
Initializing Deluge,
Starting windscribe ... OK,
usermod: no changes,
21:48:41 [ERROR   ][deluge.common:131 ] Unable to use default config directory, exiting... ([Errno 13] Permission denied: '/config/.config'),
21:48:41 [ERROR   ][deluge.common:131 ] Unable to use default config directory, exiting... ([Errno 13] Permission denied: '/config/.config'),
Configuring firewall settings,
Deluged Init,
Initializing Container,
Using VPN_USERNAME and VPN_PASSWORD to login.,
Initializing Deluge,
Starting windscribe ... OK,

I will change the config folder in Portainer right now to something local bind to see if I can get around this. But this is what I get out of the box.

Edit: Nm. Guessing trying to change the location of /config means it won't go further. So I guess this is where I'm stuck for now. But progress! :)

Kabe0 commented 4 years ago

are you using an existing config folder? I had to fix the permissions settings, which were setting the folder as a root user. Now it runs as the system user 1000:1000 so that may prevent the folder from working.

To change it you can run the following on your main system sudo chown -R 1000:1000 /config

Kabe0 commented 4 years ago

@Hukuma1 I added a fix that changes the /config folder permissions in general, but I am pretty sure your issue is related to the /config folder not having the right permissions so when deluge tries to make changes it get's an access denied.

Can you try this, run the docker container where you mount the volume to a folder that does not exist. That should force docker to create the /config folder for you, and then populate it with the proper config data.

If you want it to work with the existing folder you have then my assumtion is that your user you are logged in as is not 1000:1000. To test this, go to the terminal and type in id which will spit out your ID details. If that is the case, then set the environment variables DEL_UID and DEL_GID to the user your activly logged in as.

Hukuma1 commented 4 years ago

Sorry didn't mean to confuse anyone, I do want it to just run out of the box with the preconfigured folder. So okay, we made some progress.

Starting windscribe ... OK,
usermod: no changes,
Starting windscribe ... OK,
usermod: no changes,
Starting windscribe ... OK,
usermod: no changes,
16:12:49 [ERROR   ][deluge.ui.web.json_api     :675 ] Failed to add torrent from url http://distribution.bbb3d.renderfarming.net/video/mp4/bbb_sunflower_1080p_30fps_normal.mp4.torrent,
16:12:49 [ERROR   ][deluge.ui.web.json_api     :185 ] [Failure instance: Traceback: <class 'twisted.internet.error.DNSLookupError'>: DNS lookup failed: no results for hostname lookup: distribution.bbb3d.renderfarming.net.,
/usr/lib/python3/dist-packages/twisted/internet/_resolver.py:137:deliverResults,
/usr/lib/python3/dist-packages/twisted/internet/endpoints.py:900:resolutionComplete,
/usr/lib/python3/dist-packages/twisted/internet/defer.py:459:callback,
/usr/lib/python3/dist-packages/twisted/internet/defer.py:567:_startRunCallbacks,
--- <exception caught here> ---,
/usr/lib/python3/dist-packages/twisted/internet/defer.py:653:_runCallbacks,
/usr/lib/python3/dist-packages/twisted/internet/endpoints.py:954:startConnectionAttempts,
],
Starting windscribe ... OK,
usermod: no changes,

I can confirm it connects to Windscribe automatically. I can confirm that I can still access the Deluge WebUI once I'm connected to Windscribe with masked IP. What I'm unable to do now is download a torrent.

Noticed it's less verbose? Or is missing steps from previous image? (e.g. configuring firewall settings)

Kabe0 commented 4 years ago

Ok, so at this point, we might be dealing with a more minor issue though one that is more annoying to fix. The less verbose is not an issue per say, I have noticed that happens but it does not seem to effect the launch itself.

The fact you see the Starting windscribe multiple times might be an issue. It looks like the container is failing to stay active which might be due to something with windscribe.

Something does seem to be wrong with your dns, though that might be something that just needs a computer restart to fix as it's possible it's something to do with the docker network bridge.

You can try running dig inside the docker container to see if a result is retrieved such as dig google.com.

As a note I am running on both arm and x64 without an issue and have tested your problem without the ability to reproduce.

Kabe0 commented 4 years ago

@Hukuma1 Tried out the torrent listed in your error and was able to load it in. image

Another thought, you should take a look and make sure windscribe is not acting funning. Sometimes I have seen it not fully register (happens when constantly turning the container on and off). Run inside the docker container the command windscribe status and see if it comes up with an IP address.

Hukuma1 commented 4 years ago

Got it working!

The multiple start windscribe messages is just because I kept restarting the container to try different setups (opening ports on router, etc) I think?

So not sure why, but I guess it's a requirement to fill in DNS servers in Portainer for this to work. I put in primary 1.1.1.1 and secondary 1.0.0.1 and now it works. So if anyone has this issue, make sure you enter the DNS fields under Network in Portainer. Seems to be working just fine now. Thanks a lot! :) Will close it out as issue now resolved it seems.

Kabe0 commented 4 years ago

Awesome! glad we finally got this all resolved.

I pushed an update to try to allow logs when running as a damon (which was the reason the logs were less verbose) as python does not spit out logs when daemonised.

Hukuma1 commented 4 years ago

Awesome will update it! Thank you.

Btw I know this is the last thing you'll want to see, but if in the future could you take a look at torrent client Transmission as well? Transmission with Combustion WebUI looks so much more modern. Check out this Docker Container here for a quick run. Make sure you enable the optional combustion var: https://github.com/linuxserver/docker-transmission

Hukuma1 commented 4 years ago

Btw I pulled latest update and I see it being more verbose that Deluge had init. But it's asking for password at every restart. Is that normal behavior? Feels like I only put in password only once before no matter how many times I restarted container.

Kabe0 commented 4 years ago

Yea, the docker container does not store the cache data, so each time you restart the container, it's going to wipe any session that was active before thus the password prompt every time you restart it.

I honestly mostly use deluge because of the plugins and it works well with sonarr and radarr. The GUI is a bit lack luster, but it has a pretty good balance especially for reliability. It's easy to modify the actual torrent lib settings to allow for additional features that are sometimes not enabled on most torrent clients by default.

Transmission is also pretty good, I'll take a look at some point. I just have to decide how many I want to maintain. It might be more worth while to just extract the vpn part and let people use any docker image they want. I have seen some OpenVPN solutions, but I think their configs are way to complicated for most people to setup properly.

Hukuma1 commented 4 years ago

No worries, yeah. I figured if it weren't super hard to make a kabe0\windscribe-deluge and a kabe0\windscribe-transmission dockerhubs. But this works and rocks so you're right, probably not worth it. Thanks again for making it work. :)

Kabe0 commented 4 years ago

@Hukuma1 Pushing in an update that fixes an issue where when windscribe tries to reconnect itself it will re-assign the route the offending route that was causing the gui to stop responding.

I have made a cronjob that ensures that the line is never applied so you will want to update your image otherwise the GUI will stop working if you ever loose internet connection.

Hukuma1 commented 4 years ago

So I tried windscribe disconnect and then windscribe connect and it did not allow me to log back into the WebUI still.

Kabe0 commented 4 years ago

Oh, it's a cronjob. I can't hook into the windscribe service directly as it does not offer that, so instead every minute it runs a really smash bash script that just confirms that no address is found. So if you wait a minute, it should automatically resolve the UI issue.

It's not something that should be really noticable in the real world.

Hukuma1 commented 4 years ago

Gotcha! Nice.

Regarding config not saving after container restarts... how to keep Label plugin to stay enabled? Or speed limit parameters?

Kabe0 commented 4 years ago

Huh, odd. It should save. Make sure you hit apply and maybe try logging out once.

The file is found under /config/.config/deluge/core.conf and it looks like this when you enable a plugin

image

I wonder if it's a permission issue again? A hack would be to set the file to permissions chmod 777 core.conf to allow anyone to edit if permissions are the issue.

Hukuma1 commented 4 years ago

So odd. Now it works. Maybe the logout/login trick made it stick. Thanks!

Kabe0 commented 4 years ago

Sigh, pushing another update. Forgot to add the command to actually turn on the cronjob. Forgot it's two commands. My bad.