Docker setup

[krash1220]

I'm a little confused with how to set this up. I got docker installed okay. And I did the docker pull datashaman/putio-automator

Are we suppose to fill in the putio token and folders and then run the command?

docker run --rm -it \ -e PUTIO_TOKEN=xxxxxxx \ -p 9001:9001 \ -v /TV/Incomplete \ -v /TV/Complete \ -v /TV/Torrents \ datashaman/putio-automator

Or are we suppose to run these commands and put the info in the config.py?

RUN mkdir -p \ /app/logs /app/run \ /files/incomplete /files/downloads /files/torrents \ && touch /app/run/app.db \ && chown -R www-data /app/run /files

COPY app.py manage.py README.md requirements.txt /app/src/

RUN cd /app/src \ && pip install -U pip \ && pip install -r requirements.txt

and then run with nothing filled in?

docker run --rm -it \ -e PUTIO_TOKEN=oauth_token \ -p 9001:9001 \ -v /local/incomplete:/files/incomplete \ -v /local/downloads:/files/downloads \ -v /local/torrents:/files/torrents \ datashaman/putio-automator

Either way I can't get anything downloaded from putio.

[datashaman]

For your purposes, you ignore everything except for the docker pull and run. config.py is sorted out inside the container, and is only used by you if you wish to re-build the docker image.

So the correct set of steps for you (as a docker user) would be:

docker pull datashaman/putio-automator
docker run --rm -i \ 
    -e PUTIO_TOKEN=XXXXXX 
    -p 9001:9001 
    -v /TV/Incomplete:/files/incomplete 
    -v /TV/Complete:/files/downloads 
    -v /TV/Torrents:/files/torrents 
    datashaman/putio-automator

Replace XXXXXX with your oauth_token. With the volume declarations, the left side of : is the local (host) folder, to the right of it is the container folder. The right (container) folders must stay the same, so the app inside the container knows where to find and put things.

I'll work on making the docs clearer, specifically with the separation of the two tasks running and building the docker image.

I can also add these commands to the main CLI app and ask the user for the options interactively. That sounds like a good plan.

[krash1220]

Interactive would be great I think.

It will only upload one torrent in the torrent folder even if there's more than one but if I restart watcher from http://localhost:9001 it will upload another one and sometimes the one it already uploaded but I have to keep restarting it. And the uploaded torrents don't get deleted after upload. The downloader won't run at all. If I click on the downloader link in http://localhost:9001 I get

-EVO' Traceback (most recent call last): File "manage.py", line 213, in manager.run() File "/usr/local/lib/python2.7/dist-packages/flask_script/init.py", line 412, in run result = self.handle(sys.argv[0], sys.argv[1:]) File "/usr/local/lib/python2.7/dist-packages/flask_script/init.py", line 383, in handle res = handle(*args, *config) File "/usr/local/lib/python2.7/dist-packages/flask_script/commands.py", line 216, in call return self.run(args, *kwargs) File "manage.py", line 167, in files_download f.download(dest=app.config['INCOMPLETE'], delete_after_download=True, chunk_size=int(chunk_size)1024) File "/usr/local/lib/python2.7/dist-packages/putiopy.py", line 318, in download self._download_directory(dest, delete_after_download, chunk_size) File "/usr/local/lib/python2.7/dist-packages/putiopy.py", line 327, in _download_directory os.mkdir(dest) OSError: [Errno 13] Permission denied: '/files/incomplete/Call of the Wolf 2017.HDRip.XviD.AC3-EVO'

[datashaman]

Ok, that sounds like a permissions problem. I'll have it fixed by lunchtime (GMT+2). Gotta cover the paying gig for a while. :P

[datashaman]

Try this on your local (host) /TV folder:

sudo chmod a+rwx /TV /TV/Incomplete /TV/Complete /TV/Torrents

That allows all users on your machine to access that folder (just so you know) but it might also allow your container to write there as well.

[krash1220]

Yes, that worked. I always do sudo chown -R krash:krash on all of my data folders but that wouldn't give access to the program I guess. I'm the only one who uses this machine so all users is fine. Thank you.

[datashaman]

Good to hear! I'll move onto making it more useful now I know the basics work.

How would you use this? I see two ways: immediate download, and scheduled downloads with configurable start and end times.

All the management commands must also get a new option to work on a running docker instance (if invoked from the host), without breaking regular (non-docker) usage.

[krash1220]

Well, I'm still having permission problems. I did the sudo chmod a+rwx /TV /TV/Incomplete /TV/Complete /TV/Torrents but when the program downloads the file ownership changes to www-data and then Sickrage can't process the files because of permissions. I tried adding myself to the www-data group but that didn't work, so I tried adding www-data to my group but that didn't work. I probably screwed something up in all of that but I thought ownership couldn't be changed so easily.

I would love to have immediate download and a way to schedule it as well. I even tried to start the container as a service by reading the directions here https://docs.docker.com/engine/admin/host_integration/ but I couldn't figure out what I was doing wrong. Sometimes when I run the command to start the container I have to open a browser and manually restart the downloader. I don't know if it didn't get shutdown correctly or what but it's a pain to have to check every time. Being able to schedule certain times for downloads would be awesome.

[datashaman]

I should have a release ready (including the above) this weekend.

[krash1220]

Okay, sounds good.

[datashaman]

Please have a look at the https://github.com/datashaman/putio-automator/tree/release/1.0.0 branch. It includes new documentation, a rewrite of the CLI commands and a docker implementation that be configured from the command-line. Let me know if you run into issues with the readme and the process involved, and I'll see what I can do to smooth things over.

[datashaman]

Once we've sorted out any issues, I'll publish a full release.

[krash1220]

I love it so far. It is working like it suppose to. The only thing was when running putio docker run -s 17:44, I get errors and I had to manually start the downloader. The /usr/local/lib/python2.7/dist-packages and /usr/local/lib/python2.7/site-packages folders are empty on Mint 18.1. or at least on my machine. I added echo 'PATH=$HOME/.local/bin:$PATH' >> .profile # or .bashrc or .zshrc, ymmv to my path. The cogapp.py only exists in my home folder. Python is in /usr/lib/python2.7 but no cogapp. Do I need to add something else to my path? Like I said, it still works and downloads just fine, it just complains about it first. LOL putio docker run -s 17:44 Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/cogapp/cogapp.py", line 714, in main self.callableMain(argv) File "/usr/local/lib/python2.7/dist-packages/cogapp/cogapp.py", line 705, in callableMain self.processArguments([a]) File "/usr/local/lib/python2.7/dist-packages/cogapp/cogapp.py", line 679, in processArguments self.processOneFile(args[0]) File "/usr/local/lib/python2.7/dist-packages/cogapp/cogapp.py", line 628, in processOneFile sNewText = self.processString(sOldText, fname=sFile) File "/usr/local/lib/python2.7/dist-packages/cogapp/cogapp.py", line 567, in processString self.processFile(fOld, fNew, fname=fname) File "/usr/local/lib/python2.7/dist-packages/cogapp/cogapp.py", line 507, in processFile sGen = gen.evaluate(cog=self, globals=globals, fname=sFile) File "/usr/local/lib/python2.7/dist-packages/cogapp/cogapp.py", line 147, in evaluate eval(code, globals) File "/etc/cron.d/putio-automator+1", line 4, in Cogging /etc/cron.d/putio-automator ValueError: invalid literal for int() with base 10: '17:44'

[datashaman]

The start_hour should be the hour that the downloader must start, not the specific time. So in my case, I get cheaper downloads from midnight to 6AM, so I would use this to run it:

putio docker run -s 0 -e 6

The downloader is not automatically started when the container is launched. It starts at 0 past the start_hour, and ends at 0 past the end_hour. During that period, it will re-start (ie check for new files) every few minutes (only between start_hour and end_hour), specified with -c or --check_downloads_every.

If both start_hour and end_hour are the same (24 hour operation), it just adds a line to re-start every few minutes (using -c config).

[krash1220]

Okay, I tried again. I ran putio docker run -s 13 -e 6 which is 1 PM and stops at 6 AM. Downloader still did not start.

Also, the files are dowloaded into /TV/Incomplete but aren't being moved to /TV/Complete. I have to move them manually.

[krash1220]

I don't think it's creating the cron job.

I ran sudo docker run --rm -i -e START_HOUR=15 -e END_HOUR=16 -e CHECK_DOWNLOADS=15 -e PUTIO_TOKEN=XXXXXXXX -p 9001:9001 -v /local/incomplete:/TV/Incomplete -v /local/downloads:/TV/Complete -v /local/torrents:/TV/Torrents datashaman/putio-automator to see if it was a permissions problem but still didn't work. In the terminal it states Cogging /etc/cron.d/putio-automator (changed) but there is no /etc/cron.d./putio-automator /etc/cron.d is empty. It's not creating the putio-automator job?

[datashaman]

Strange one, it might be that you have an old image, try:

putio docker pull

You should not be running docker as root. This might be source of why you cannot run the usual putio docker run. Docker is usually setup with access permissions requiring membership in the docker group.

Try doing this:

sudo usermod -a -G docker yourusername

Or:

sudo adduser yourusername docker

Then log out and in again so the group membership takes effect. Check it's correct and you can see the group when you run this:

id 

[krash1220]

I just ran docker as root to see if it would start the cron job. I already did the sudo usermod -a -G docker krash and logged out and in. The only reason I ran it as root was because I couldn't run putio dock run as root. Either way, no matter what I do, the cron job isn't being created for some reason and the download never starts. Oh and I already did the putio docker pull, so I have the latest. I just ran the full docker command like putio would have to see if it would work as root.

[datashaman]

Please send me the IMAGE_ID you are using:

docker images datashaman/putio-automator:latest

[krash1220]

a7a8eb8b08c0

sorry, I just did another docker pull and now it's saying 459ba8de6ef2

[datashaman]

Please run:

putio docker pull

My current output is this:

Using default tag: latest
latest: Pulling from datashaman/putio-automator
Digest: sha256:d067761ac92dae4daa65657bc6a89e2d3b9f0f38629f514f090161a8e9ad5bbf
Status: Image is up to date for datashaman/putio-automator:latest

[datashaman]

Also, update the pip installation to the latest release candidate branch (not released yet):

pip install -U --user git+https://github.com/datashaman/putio-automator@release/1.0.0

[datashaman]

My latest IMAGE_ID matches yours now. Upgrade the python package as well, and see how that works.

[krash1220]

Well, that did it. I did the putio docker pull and the upgraded python. The strange part is I ran putio docker run -s 0 -e 23, which is midnight correct? and 11 PM because at the moment I have a lot of files to download but the downloader started already. Not that it's a problem because I'm in the states and I don't have to worry about downloading at certain times. But if I had wanted it to start then it would have been. Is there something I'm missing? I'll retry it and use -s 0 -e 6 just to see if it still starts right away. Okay, that time span didn't make it start right away but it doesn't make sense to me but it's working so that's all that matters. LOL

[datashaman]

23 is 11PM, if you want it to run continuously use: -s 0 -e 0. It will start after -c minutes.

[datashaman]

A recent change means you must setup some more envvars (if you are running manually), see the code here: https://github.com/datashaman/putio-automator/blob/release/1.0.0/putio_automator/commands/docker.py#L61

-e DOWNLOADS=/files/downloads
-e INCOMPLETE=/files/incomplete
-e TORRENTS=/files/torrents

This also works:

putio docker run

[krash1220]

It's broken. I did a putio docker pull and go the new image

Using default tag: latest latest: Pulling from datashaman/putio-automator Digest: sha256:fa5d96a188dfb81c126f9aed0fd7f89d53fedb2e908960d18ebadda589c92872 Status: Image is up to date for datashaman/putio-automator:latest

opened supervisor status in browser refresh restart all stop all no programs to manage

I ran putio docker run still blank putio docker run -s 0 -e 23 still blank putio docker run -s 0 -e 23 -e DOWNLOADS=/TV/Complete -e INCOMPLETE=/TV/Incomplete -e TORRENTS=/TV/Torrents still blank

Had to run putio files download putio torrents watch -a and that works.

[datashaman]

Remember to also update the python package: https://github.com/datashaman/putio-automator/issues/7#issuecomment-282652667

And double-check my docker-run task inside Makefile, which usually has all the right command line options to get it running. Sorry things are moving around quite fast at the moment. I'm ironing out a lot of small usability issues. Especially note some of the new env vars, and also set TZ to something like 'Africa/Johannesburg' or 'Australia/Adelaide'. Complete list is here: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones

[krash1220]

Okay, now I'm totally confused. LOL I updated the python package like before, I also saw the new env vars which is why I tried putio docker run and then putio docker run -s 0 -e 23 -e DOWNLOADS=/TV/Complete -e INCOMPLETE=/TV/Incomplete -e TORRENTS=/TV/Torrents still didn't work. I'm not sure about the TZ part. Do you mean I have to add your time zone to my computer in order to run the program?

[datashaman]

Yikes, I uncovered an issue with changes I made to the pkg recently. Please upgrade both pkg and docker image and it should be back to normal again. I'm going to put some solid tests in place so it doesn't slip again. I've been putting a lot of work into a Raspberry PI image which has caused a bit of flux in the dev process. Apologies!

[datashaman]

As before, check Makefile task docker-run for the manual invocation or use putio docker run

[datashaman]

You should set -e TZ=blah where blah is your local timezone (from that list). This will allow the schedule to know what exact time you want things to start and end.

[datashaman]

It won't make any difference to you if you have -s 0 -e 0 since the downloader will run continuously.

[krash1220]

not working pip install -U --user git+https://github.com/datashaman/putio-automator@release/1.0.0 putio docker pull putio docker run -s 0 -e 0 http://localhost:9001/ This site can’t be reached

[datashaman]

Oops, I forgot to change the port number in the docker run command. I've pushed up a change that fixes this so it uses -p 9001:80, which is now the correct way to map ports. Please confirm it now works, and I think we can close this ticket for now. :)

[krash1220]

Yes, that did it. All is working great. Thanks.

[datashaman]

Sweet! I'll finish the release just now, so a regular pip install will work just fine from now on.

[datashaman]

So to upgrade from now on, use this:

 pip install -U --user putio-automator

And then:

putio docker pull

The release branch is integrated into the master branch now, so it no longer exists.