mikedhanson commented 1 year ago

Any thoughts on running this from docker?

I am an avid unraid user and think that having this run via docker would make sense. Dockerfile could look something like this?

requirements.txt

spot_sync

dockerfile

#https://docs.docker.com/language/python/build-images/ 

#base image
FROM python:3.8-slim-buster

#Set Environment Vars for docker
ENV ConfigPath = **None** \
    DownloadPath = **None** \
    arl = **None** \
    SpotID = **None** \ 
    SpotSecret = **None** \ 
    SpotRedirect = **None** \ 
    DiscordBotToken = **None** \ 
    CHANNEL = **None** 

WORKDIR /usr/src/app
COPY requirements.txt ./

# install spot_sync from requirements 
RUN pip3 install --no-cache-dir --upgrade pip && pip install --no-cache-dir -r requirements.txt

COPY . .

# cmd to run container at start
CMD [ "python3", "./SpotifyRemix.py" ]

danielcharrua commented 1 year ago

Wow this would be great to see in this package! Any thoughts @jbh-cloud?

jbh-cloud commented 1 year ago

@danielcharrua the main reason I've held off on this is that currently spotify_sync stores all of its state in user data. This includes which songs have been matched, downloaded etc. Adding docker to the mix would warrant the need to do bind volume mounting to ensure it still access to this from the host which kinda negates the transience of docker. Adding configuration into the mix (as a bind volume) and potentially supporting profiles would also add complexity to this.

It's possible to refactor the code base to support this configuration via environment variables etc. but would require another way to validate parameters (currently using json schema).

I'm not opposed to adding docker support but I'm not sure I understand the requirements / use case (apart from specifically running on a NAS).

Happy for anyone with an idea of how they would expect to use it to chime in :)

erstert3st commented 1 year ago

use cases: running on singleboard computer (Pi) fast easy deploying fits well with "set and forget" codebase for future projects
:)

danielcharrua commented 1 year ago

Yes, also doing a daily/weekly backup of your library on a NAS running docker would be a great one. I use docker for many other backups on services, really easy, plug & play and forget.

erstert3st commented 1 year ago

I have a working Dockerfile last question - for what do you need ? -or better question where to set them? DiscordBotToken = None \ CHANNEL = None

erstert3st commented 1 year ago

https://github.com/jbh-cloud/spotify_sync/pull/21

testet on ubuntu

jbh-cloud commented 1 year ago

@erstert3st first of all, thanks for you work on this! I guess it kind of brings me back to my original usecase question. Are we wanting to replicate the exact functionality of it but just allow it to be run via docker or do we want to extend it?

Your implementation is designed to run in docker as a continuous process (while True loop) against a specific config file that is built into the image. As it stands spotify_sync is designed to be a one-shot application as opposed to long running instance. I think I would prefer a more bare-bones setup where spotify_sync is the docker entrypoint allowing power users more flexibity if they chose.

E.g

Cache spotify oauth token once (could even do this from any other machine and put the file in the right directory instead)

docker run \
   -it \ # so that we can provide oauth authorization
   --rm \ # we dont want this hanging around
   -v /mnt/local/downloads:/downloads \ # can map this to whatever you want, needs to be consistent with config.json
   -v /mnt/local/spotify_sync_pd:/root/.config/spotify_sync \ # This could even be left out if you didn't care about losing which songs had been processed..
   -v /mnt/local/spotify_sync_configs:/configs \ # One or more configs to pass through to container for use with `--config` or `--profile`
   --name spotify_sync \
   spotify_sync:latest \
   utils authorize-spotify --config /configs/user1.json

The run it..

docker run \
   --rm \ 
   ...
   spotify_sync:latest \
   run auto --config /configs/user1.json

If people are fine with this, it means no code change (apart from a Dockerfile and no needing to maintain a separate docker script / runtime etc. This would also mean you could schedule it just as before using cron.

erstert3st commented 1 year ago

first the loop thing, yea i understand that point, I need to run it in a loop for my project but your solution is better because if somebody needs to run in a loop he can loop the container anyway

The other thing, I'm fine with this but it's not the Docker way, to depend on an external file that is needed to work. Normally you would set the envVars in Dockerfile/Composer... and that's it, maybe load them into config or into configCreatingProcess(like mine commit) or use your normal config creating thing and then replace the Values with Python or shell script but I don't know how the authentification thing work so yea other short question: How much work do you think is it to download each playlist into their separate Folder- same structure as yours except that the root has the playlist name/ Big Thanks for your project :D btw hope you understand my English 😅

jbh-cloud commented 1 year ago

Sorry I have been really busy and haven't had much time. First of all, I completely agree that volume bind mounts pretty much go against the whole point of docker. I would love to support completely argument parsed configuration. However this is currently hindered by:

Complex data structures as environment variables, how would we pass our list of excluded playlists to the app via var without doing something gross like a json string
Oauth, it would be super easy to do this if we didn't need to authenicate to Spotify with a user grant. In order for spotifysync to access your private_ songs and playlists it must fetch and store this token. Without it, you would need to 'reauthorize' the app every run (which isn't automatic in it's current state).

I have some thoughts about how this may work going forward but would be a significant rewrite to faciliate.

In the interim, here is a fully working example running it via volume mounts. It also allows you to access all of the subcommands (as these are just passed in as paramters to the entrypoint...

FROM python:3.10-slim-buster

ARG VERSION

WORKDIR /app

RUN pip3 install --no-cache-dir --upgrade pip && pip install spot_sync==${VERSION} --no-cache-dir

ENTRYPOINT ["spotify_sync"]

# Suuuper basic scratch dockerfile workaround for people wanting to test docker functionality
# This allows you to run with volume binds without losing access to (primarily) OAuth token

# Build image with:
# docker build --build-arg VERSION=1.1.1 --pull --rm -f Dockerfile -t spotify-sync:1.1.1 . --no-cache

# Run

# Assumes the following:
# $(pwd)/cfgs exists and contains a config.json with a download path of /downloads
# $(pwd)/dl exists and is empty
# $(pwd)/pd exists and is empty

# Cache oauth token..
# docker run -it -v $(pwd)/cfgs:/configs -v :/downloads -v $(pwd)/pd:/root/.local/share/spotify_sync/ spotify-sync:1.1.1 utils authorize-spotify --config /configs/config.json
# You will see that we have some important data in $(pwd)/pd
# Run in auto
# docker run -it -v $(pwd)/cfgs:/configs -v $(pwd)/dl:/downloads -v $(pwd)/pd:/root/.local/share/spotify_sync/ spotify-sync:1.1.1 run auto --config /configs/config.json

KINOTheProducer commented 1 week ago

Apologies for the long writeup and very janky way of making this work, I am not an expert on Docker and sort of threw this together while reading lots of documentation, but it works. :)

I now have this working in Unraid now as a User Script running on a schedule.

Here's the breakdown of how I did it:

I cloned the repository to my appdata to start, this is the command I used:

git clone https://github.com/b4ird/spotify_sync.git

Then, in the same folder, I added a Dockerfile and a requirements.txt file

Dockerfile:

FROM python:3.9-slim

WORKDIR /app

COPY . /app

RUN pip install --no-cache-dir -r /app/requirements.txt

RUN python3 -m pip install -U spot_sync

CMD ["/bin/bash"]

Requirements.txt:

click==8.1.3
click_option_group==0.5.6
attrs==22.2.0
tabulate==0.9.0
spotipy==2.19.0
psutil==5.9.0
jsonschema==4.17.3
flatdict==4.0.0
appdirs==1.4.4

Building the docker image we just made in the terminal:

docker build -t spotify-sync .

First run (auth, initial bulk song saving, etc)

Next, we'll run the docker image in a terminal. This will be run in interactive mode so we can do the initial setup. We will also mount a few folders so that the information persists for future runs:

docker run -it   
-v /mnt/user/appdata/spotify_sync:/app   
-v /mnt/user/appdata/spotify_sync/config:/app/config   
-v /mnt/user/data/media/music:/music   
-v /mnt/user/appdata/spotify_sync/profiles:/root/.local/share/spotify_sync   spotify-sync

The first thing we'll do is create a starting point for our config file by using the command in the Quick Start guide:

spotify_sync config generate

Make sure to go in and edit your newly created config.json, adding in all the info that is relevant to you. In this example, we will make the download directory /music

Then create a profile, replace myFirstProfile with whatever you want it to be named:

spotify_sync config add myFirstProfile ./config.json

Create a Spotify app, as outlined in the docs, and then go through with the Spotify authentication:

spotify_sync utils authorize-spotify --profile myFirstProfile

Now you're ready to run the download script for the first time. Note: this initial download took me about 4 hours (but I have 14,000 songs saved)

spotify_sync run auto --profile myFirstProfile

Once this one-time saving of your initial library is complete, you can go ahead and make this an automated script for future purposes (where you will be adding only new songs to your library)

User Script Setup (scheduling future runs)

First, you'll need the User Scripts plugin from the Community Applications.

Once you have that, you'll make a new script. I named mine Spotify to Plex Sync. You'll go in and edit the script and paste this in:

#!/bin/bash

# Remove any existing container with this name
docker rm -f spotify-sync 2>/dev/null || true

# Run the Spotify sync commands
docker run --name spotify-sync \
  -v /mnt/user/appdata/spotify_sync:/app \
  -v /mnt/user/appdata/spotify_sync/config:/app/config \
  -v /mnt/user/data/media/music:/music \
  -v /mnt/user/appdata/spotify_sync/profiles:/root/.local/share/spotify_sync \
  spotify-sync bash -c "spotify_sync config add myFirstProfile /app/config.json && spotify_sync run auto --profile myFirstProfile"

  # Remove the container after it completes
docker rm -f spotify-sync

Essentially, this does a few things: it first removes any instance of the spotify-sync image, so there are no conflicts, then it mounts the previous folders we need within the app so that the saved state of already downloaded songs persists. Then it is running two commands, the first is latching the config to our profile and the second runs spotify_sync in auto mode with that profile.

Save your changes once you add this script, and make the cron on it whatever you'd like. I have it set to 0 which makes it run at the top of every hour.

Hit Apply at the bottom of the page and your script should run automatically from now on! 🎉

b4ird / spotify_sync

Docker support? #17

testet on ubuntu

Dockerfile:

Requirements.txt:

First run (auth, initial bulk song saving, etc)

User Script Setup (scheduling future runs)