search

netromdk / slacker

Slacker - Easy access to the Slack API and admin of workspaces/teams.
MIT License
14 stars 0 forks source link

Feature/concurrent downloads #110

Closed 0verbyte closed 5 years ago

0verbyte commented 5 years ago

Wanted to get some initial feedback on this.

Changes proposed in this pull request:

This isn't quite ready to be merged since the save file after the downloads have finished isn't being generated. There is now weirdness with the logger, since things are running concurrently the main REPL will return immediately after calling execute_tasks, and logs will start showing up afterwards moving the Slacker > prompt out of context. Will need to decide how to best reset the prompt.

0verbyte commented 5 years ago

Travis is failing because __write_save_file is unused now, will fix.

netromdk commented 5 years ago

Regarding the log output, it might be best to buffer that until it either succeeds or fails, and then post to the actual logger? Then it won't mess with the stdout until ready/finished.

0verbyte commented 5 years ago

When I run it as-is with emoji.list -p /tmp/ then I get a lot of these exceptions

Interesting that you're getting those exceptions. It seems to be related to Python version 3.6.5, I'm running 3.7.0 and not seeing those exceptions. I'll investigate further.

Regarding the log output, it might be best to buffer that until it either succeeds or fails, and then post to the actual logger? Then it won't mess with the stdout until ready/finished.

The REPL is now blocked until the tasks have completed. Blocking should be fine in this case since a user shouldn't be able to execute multiple commands while others are running, that could get messy IMO.

Although it could be interesting to support optional multiple command execution as a command line multi-flag, such as --command [COMMAND] --command [COMMAND].

% ./slacker.py --command emoji.list -p /tmp/emojis/ --command channels.list
0verbyte commented 5 years ago

I was able to repro the issue with Python 3.6 running in a Docker container.

FROM python:3.6-alpine

WORKDIR /app/slacker/
COPY . .

RUN pip install -r requirements.txt

ENTRYPOINT ["python3", "slacker.py"]

Instead of passing the logger directly to the tasks, the logs are being buffered and the caller can choose to replay the logs or not. This seems to fix the issue where things weren't able to be locked, can you give it a try now?

netromdk commented 5 years ago

Now it works great! Thanks!