Self hosting - Githubissues

mpapierski commented 6 years ago

Hi!

I recently started to use TaskWarrior after moving all my tasks from Things.app. I saw this project while looking for iOS client and looks very neat! I was trying to self host it, but I noticed it won't be that easy:

Hard coded paths in settings.py
memcached/redis addresses seem to be hardcoded
Not clear which services should be ran, and how

I saw there are at least 3 services required:

Database
Memcached
Redis

Daemons:

Celery workers
Web server (uwsgi, but perhaps gunicorn will work too)
Ember daemon?
taskd?

I was wondering if it is on your roadmap to add support to make self hosting easier?

coddingtonbear commented 6 years ago

I'm afraid I don't have anything like that on my roadmap! I have such a very small amount of time available for maintaining Inthe.AM week-to-week that it's not likely that I'll ever be able to contribute toward helping figure out how to self-host.

That being said, I do think self-hosting is a great idea, and am definitely available to help out by answering questions -- feel free to ping me in the gitter channel. I think there might even be a few others who have asked about that in other issues, if you were looking for folks to collaborate with.

If I were to recommend a starting point: there are vagrant configs in the repo that will configure a virtual machine as a dev environment. They don't create an environment that looks much at all like production (in case it isn't clear: definitely don't run the vagrant environment on a public-facing server -- doing so would be extremely risky), but they might give you a sense of what services are involved.

Normally, I'd close a ticket like this since it's not actionable for me, but I think I'll leave this one open in in case others might want to chime in to offer you some help in figuring out how to self-host. Good luck!

n1ckPrince commented 6 years ago

This is great! I’d love to see the energy go toward self-hosting. The integrations with Trello and Calendar really make this a rad project.

coddingtonbear commented 6 years ago

I'm afraid my energy is already allocated just to keeping things chooching as they are, but good luck on finding other folks to help you out in sorting out self-hosting!

aquaplanet commented 6 years ago

I am by no means a good programmer or have alot of energy or time, but this looks like a good thing I really want. No promises at all but I might look into it.

westurner commented 6 years ago

A Dockerfile that calls the current vagrant init would be useful (and more up to date than the precise32 which the Vagrantfile currently builds from)
docker-compose.yml would be useful for development and also for generating Kubernetes Pod JSON/YML.

Are there a few constants [that should/could be] in settings.py that need to be changed to support self-hosting? a 12-factor configuration may solve for these additional variables.

django-environ does 12-factor configuration from environment variables
- https://github.com/joke2k/django-environ
https://github.com/pydanny/cookiecutter-django#features
- docker-compose config w/ celery, redis, postgresql
- django-environ
- https://github.com/pydanny/cookiecutter-django/tree/master/%7B%7Bcookiecutter.project_slug%7D%7D/compose/local/django
- https://github.com/pydanny/cookiecutter-django/tree/master/%7B%7Bcookiecutter.project_slug%7D%7D/compose/production

A Dockerfile can just call shell scripts and checkpoint after, or e.g. call parts of shell scripts and checkpoint in between operations (in order to skip the long package-downloading-and-building bootstrap steps).

westurner commented 6 years ago

Here's how to run the taskd (and celery) daemon processes with systemd instead of upstart: https://taskwarrior.org/docs/taskserver/control.html#systemd

beatgammit commented 6 years ago

I might have time to work on this soon. Here's my plan of attack:

document how to build it (in the wiki for now)
make a docker image that starts the server and databases (not necessarily functional)
make a list of known things that are broken with self hosting and update this issue with it
make a pull request with changes so far
work on getting a functional self hosted app on my server

I'm not sure how to best break it up if someone wants to help, but I think once I get a docker image running, people could hopefully jump in and help out.

There's a decent chance I can get started this weekend, but that's not a promise at all. ;)

mpapierski commented 6 years ago

@beatgammit Sounds like a good plan!

What do you think about supporting docker-compose.yml? This seems like good approach since multiple components are involved. Also I'd suggest splitting database part out of the server image.

As @westurner suggested it might be a good start to move vagrant init script into Dockerfile and see what will happen.

beatgammit commented 6 years ago

@mpapierski

Yeah, that sounds like a plan. I do a bit with docker-compose already, so I can probably throw something together reasonably quickly, though I don't know how involved the app is to start. I also don't know too much about Vagrant, but it seems the config here is pretty simple.

It looks like it's also using an Ubuntu 12.04 base, so I'll probably start building off that, but I'd like to strip it down to an Alpine container if possible, though just updating to the latest Ubuntu is also a win.

I'll work in my own fork until I get something going. I might have some time tomorrow, but no promises.

coddingtonbear commented 6 years ago

If I were you, I'd recommend building the environment off of the latest LTS stable Ubuntu image rather than 12.04; I don't think there are even security releases for 12.04 these days, and the production environment isn't even still on 12.04 (14.04 for now).

westurner commented 6 years ago

To convert docker-compose.yml (useful for developers) to Kubernetes resources (particuarly useful for deployment) -

westurner commented 6 years ago

I'd definitely upgrade to latest LTS. apt package names may have changed?

jeaye commented 6 years ago

This would be superb, guys; thanks for looking into it!

beatgammit commented 6 years ago

Is there a particular advantage of kubernetes over a simple docker image for this particular deployment? I doubt it'll need to scale too much...

lanefu commented 6 years ago

@beatgammit howd it go?

westurner commented 6 years ago

Is there a particular advantage of kubernetes over a simple docker image for this particular deployment? I doubt it'll need to scale too much...

Systemd can re-spawn docker-compose when/if it/they halts.

https://docs.docker.com/config/containers/start-containers-automatically/

A restart policy only takes effect after a container starts successfully. In this case, starting successfully means that the container is up for at least 10 seconds and Docker has started monitoring it. This prevents a container which does not start at all from going into a restart loop.

Warning: Do not try to combine Docker restart policies with host-level process managers, because this creates conflicts.

Kubernetes will also handle that (with health checks).

https://kubernetes.io/docs/tutorials/k8s201/#health-checking

https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/

westurner commented 6 years ago

For example, liveness probes could catch a deadlock, where an application is running, but unable to make progress. Restarting a Container in such a state can help to make the application more available despite bugs.

The kubelet uses readiness probes to know when a Container is ready to start accepting traffic. A Pod is considered ready when all of its Containers are ready. One use of this signal is to control which Pods are used as backends for Services. When a Pod is not ready, it is removed from Service load balancers.

lanefu commented 6 years ago

Nomad can do that too :P

....But step 1 would be actually breaking things out into discrete services, identify their dependencies and respective configuration needs.

bertiebaggio commented 5 years ago

I might have time to work on this soon. Here's my plan of ... There's a decent chance I can get started this weekend, but that's not a promise at all. ;)

@beatgammit did you make any progress on this, out of interest?

renefritze commented 4 years ago

So I've started to work on getting a docker-compose setup for all the involved services. I'm at a point now though where I'm confused about who calls what and what serves which suburl. It would help greatly if you could share your webserver config, @coddingtonbear or just shortly describe which part of the repo should handle what requests.

coddingtonbear commented 4 years ago

Actually— one of the things I was thinking of doing this weekend was dockerizing intheam; maybe we could just meet in the gitter channel later today to coordinate?

On May 23, 2020, at 05:34, René Fritze notifications@github.com wrote:

So I've started to work on getting a docker-compose setup for all the involved services. I'm at a point now though where I'm confused about who calls what and what serves which suburl. It would help greatly if you could share your webserver config, @coddingtonbear or just shortly describe which part of the repo should handle what requests.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

nodeselector commented 4 years ago

👋 I don't see that the containerization occurred. Is this because of some technical hurdle or pure time constraints? Either way, I have some time to help.

coddingtonbear commented 4 years ago

I have something that might possibly work in the dockerize branch, but I haven't found the time yet for bringing it across the finish line and could definitely use the help. Feel free to send me a message on the gitter.im channel & I can talk you through the current known and unconfirmed problems, @nodeselector.

coddingtonbear commented 3 years ago

There is now an easily-uppable docker environment available. Have a look at the developer docs on https://intheam.readthedocs.io/en/latest/development/index.html.

Cheers & good luck!

westurner commented 3 years ago

:+1:

https://github.com/coddingtonbear/inthe.am/blob/development/docker-compose.yml
https://github.com/coddingtonbear/inthe.am/tree/development/docker
re: the old vagrant build and tests w/ travis or GH actions (and cookiecutter-django's scripts for same) https://github.com/coddingtonbear/inthe.am/commit/332f68f1f2bb1fe8e3815ebff84cee3b4f9f15db#commitcomment-44866313
It should be possible to create a 1-node k8s cluster for this docker-compose.yml with {kompose, k3d (k3s (k8s))}

So https://inthe.am/ is a webapp and task queue built on Django + Postgres + Redis + Celery with:

uuid.uuid4() account ids
inbound email w/ required allow list: eb078fa5-07fe-4d43-9a0b-5e5ffb59f89b@inthe.am
inbound SMS (twilio: phone number and per-txt costs)
iCal
- https://inthe.am/api/v1/task/ical/due/eb078fa5-07fe-4d43-9a0b-5e5ffb59f89b
- https://inthe.am/api/v1/task/ical/waiting/eb078fa5-07fe-4d43-9a0b-5e5ffb59f89b
RSS
- https://inthe.am/api/v1/task/feed/eb078fa5-07fe-4d43-9a0b-5e5ffb59f89b
Trello boards (2 way sync)
Zapier integrations configured for the https://inthe.am install of this docker-compose.yml

2 way sync with the taskwarrior CLI app (task) with per-user SSL certs for taskwarrior client config in ~/.taskrc. From https://inthe.am/getting-started :

# Add the following settings to your Taskwarrior configuration file (~/.taskrc).
taskd.certificate=/path/to/private.certificate.pem
taskd.key=/path/to/private.key.pem
taskd.ca=/path/to/ca.cert.pem
taskd.server=inthe.am:53589
taskd.credentials=inthe_am/_username_/eb078fa5-07fe-4d43-9a0b-5e5ffb59f89b
taskd.trust=ignore hostname

coddingtonbear / inthe.am

Self hosting #291