My fork might be of interest

[hamstah]

Hi,

Sent you an email a few weeks ago as I started working on my fork, but not sure you received it. I've heavily changed your original code for my needs, and I thought it might be worth discussing how we could work together to make more people benefit from them.

My fork is at https://github.com/hamstah/clouddream

Some of the changes I introduced

• There is a web based manager that allows the uploading of new images by url/file upload
• You can browse the input/output of the file from there
• You can process any of the input images with parameters specified by the user (maxwidth, n_inter, n_octave etc), so you can have multiple outputs from the same image with different parameters
• Processing is ordered and done by workers, you can see the queue and jobs in progress (you can have multiple workers in parallel). I had issues with the original project where some image would fail and it would get stuck in a loop trying to process them.
• Use of docker compose to make it easier to spin the containers and scale the number of workers

It's obviously work in progress and I'm no docker expert, but I think it provides a good foundation to extend.

I'm thinking of moving the project out of the fork and into its own repo, as it's almost a complete rewrite.

Would be great to hear your thoughts.

I have a version of it running on a digital ocean droplet, I can give you access if you want to check it out.

[itsTeknas]

I tried your fork on google cloud, I got some errors... pls look into it Sorry had to do this here, your repository doesn't have a Issues tab.

root@dreamer:/home/blackcurrant_apps/clouddream# docker-compose up Recreating clouddream_redis_1... Recreating clouddream_manager_1... Recreating clouddream_compute_1... Recreating clouddream_nginx_1... Attaching to clouddream_redis_1, clouddream_manager_1, clouddream_compute_1, clouddream_nginx_1 redis1 | ._
redis1 | .-__ ''-._ redis_1 | _.- .. ''-. Redis 3.0.3 (00000000/0) 64 bit redis1 | .-`.-.\/ ., ''-.
redis_1 | ( ' , .-|, ) Running in standalone mode redis1 | |`-.-...- ...-.-._|'_.-'| Port: 6379 redis1 | |`-. ._ / _.-' | PID: 1 redis_1 |-. `-.-./ _.-' _.-' redis_1 | |-.`-.-.**.-' _.-'_.-'| redis_1 | |-.`-. .-'.-' | http://redis.io
redis1 | `-.-._-..-'.-' .-'
redis1 | |`-.-._-..-' .-'.-'|
redis1 | | `-.-._ _.-'_.-' | redis_1 |-.`-.-.**.-'_.-' _.-' redis_1 |-._ -.__.-' _.-' redis_1 |-. .-'
redis_1 | `-..-'
redis_1 | redis_1 | 1:M 09 Aug 16:40:11.947 # Server started, Redis version 3.0.3 redis_1 | 1:M 09 Aug 16:40:11.969 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect. redis_1 | 1:M 09 Aug 16:40:11.969 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled. redis_1 | 1:M 09 Aug 16:40:11.969 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128. redis_1 | 1:M 09 Aug 16:40:11.970 * DB loaded from append only file: 0.001 seconds redis_1 | 1:M 09 Aug 16:40:11.970 * The server is now ready to accept connections on port 6379 compute_1 | 16:40:15 RQ worker u'rq:worker:4d7af1871d2d.11' started, version 0.5.4 compute_1 | 16:40:15 compute_1 | 16:40:15 * Listening on compute... manager_1 | [2015-08-09 16:40:15 +0000] [1] [INFO] Starting gunicorn 19.3.0 manager_1 | [2015-08-09 16:40:15 +0000] [1] [INFO] Listening at: http://0.0.0.0:8000 (1) manager_1 | [2015-08-09 16:40:15 +0000] [1] [INFO] Using worker: sync manager_1 | [2015-08-09 16:40:15 +0000] [14] [INFO] Booting worker with pid: 14 manager_1 | [2015-08-09 16:40:18 +0000] [14] [ERROR] Exception in worker process: manager_1 | Traceback (most recent call last): manager_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 507, in spawn_worker manager_1 | worker.init_process() manager_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 118, in init_process manager_1 | self.wsgi = self.app.wsgi() manager_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/base.py", line 67, in wsgi manager_1 | self.callable = self.load() manager_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 65, in load manager_1 | return self.load_wsgiapp() manager_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp manager_1 | return util.import_app(self.app_uri) manager_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/util.py", line 355, in import_app manager_1 | import(module) manager_1 | File "/opt/manager/manager/init.py", line 1, in manager_1 | from app import app manager_1 | File "/opt/manager/manager/app.py", line 18, in manager_1 | from models import ( manager_1 | File "/opt/manager/manager/models.py", line 7, in manager_1 | from utils import create_thumbnail manager_1 | ImportError: No module named utils manager_1 | Traceback (most recent call last): manager_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 507, in spawn_worker manager_1 | worker.init_process() manager_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 118, in init_process manager_1 | self.wsgi = self.app.wsgi() manager_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/base.py", line 67, in wsgi manager_1 | self.callable = self.load() manager_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 65, in load Attaching to clouddream_redis_1, clouddream_manager_1, clouddream_compute_1, clouddream_nginx_1 manager_1 | return self.load_wsgiapp() manager_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp manager_1 | return util.import_app(self.app_uri) manager_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/util.py", line 355, in import_app manager_1 | import(module) manager_1 | File "/opt/manager/manager/init**.py", line 1, in manager_1 | from app import app manager_1 | File "/opt/manager/manager/app.py", line 18, in manager_1 | from models import ( manager_1 | File "/opt/manager/manager/models.py", line 7, in manager_1 | from utils import create_thumbnail manager_1 | ImportError: No module named utils manager_1 | [2015-08-09 16:40:18 +0000] [14] [INFO] Worker exiting (pid: 14) manager_1 | [2015-08-09 16:40:18 +0000] [1] [INFO] Shutting down: Master manager_1 | [2015-08-09 16:40:18 +0000] [1] [INFO] Reason: Worker failed to boot. clouddream_manager_1 exited with code 3 Gracefully stopping... (press Ctrl+C again to force) Stopping clouddream_nginx_1... Stopping clouddream_compute_1... Stopping clouddream_redis_1... Exception in thread Thread-1 (most likely raised during interpreter shutdown): Traceback (most recent call last): File "/code/build/docker-compose/out00-PYZ.pyz/threading", line 810, in __bootstrap_inner File "/code/build/docker-compose/out00-PYZ.pyz/threading", line 763, in run File "/code/build/docker-compose/out00-PYZ.pyz/compose.cli.multiplexer", line 41, in _enqueue_output File "/code/build/docker-compose/out00-PYZ.pyz/compose.cli.log_printer", line 62, in _make_log_generator File "/code/build/docker-compose/out00-PYZ.pyz/compose.container", line 155, in wait File "/code/build/docker-compose/out00-PYZ.pyz/docker.utils.decorators", line 20, in wrapped File "/code/build/docker-compose/out00-PYZ.pyz/docker.client", line 826, in wait File "/code/build/docker-compose/out00-PYZ.pyz/docker.clientbase", line 83, in _post File "/code/build/docker-compose/out00-PYZ.pyz/requests.sessions", line 508, in post File "/code/build/docker-compose/out00-PYZ.pyz/requests.sessions", line 465, in request File "/code/build/docker-compose/out00-PYZ.pyz/requests.sessions", line 573, in send File "/code/build/docker-compose/out00-PYZ.pyz/requests.adapters", line 414, in send <type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'error' root@dreamer:/home/blackcurrant_apps/clouddream#

[hamstah]

Hey

Thanks for trying it, I forgot to add a file, which I've now added. Also added issues to the repo.

[itsTeknas]

@hamstah Great ! will try again !