This is an issue just meant to document some development.
Pipes is a single sinatra application. The same ruby process is responsible for getting the request, doing the work, fetching external data, running the pipe, and finally painting the HTML (and sometimes XML/JSON). That means issues in the backend (like a pipe blocking) can take down the whole site.
Solution sounds simple: Split it up. Have a backend that runs the pipes, and a frontend that delivers the results. That way, the backend could be scaled up horizontally easily (no session issues there anymore!). And it also could make it easier to add something pipes is missing: a push infrastructure, so pipes can run for example when an input feed gets an update.
So I built this. The prototype was to have a pipe_runner.rb script that gets called by the frontend, which can run a pipe by just getting its json representation (which we have in the database anyway). A full system could then queue this or use it in whatever form ideal.
The performance was too bad to continue. I seems the overhead of splitting frontend and backend - maybe the IPC, maybe missed issues with caching, I could not pinpoint it - made the whole system a lot slower. There was no way this could work on production.
Maybe the monolith is the right approach despite the issues described above. This re-architecture could also just need a different approach - maybe the backend needs to be a long-running server process again, maybe in combination with other performance improvements.
Just recording results, I will close this issue immediately as additional worked in that direction is not planned for now, though input is welcome.
This is an issue just meant to document some development.
Pipes is a single sinatra application. The same ruby process is responsible for getting the request, doing the work, fetching external data, running the pipe, and finally painting the HTML (and sometimes XML/JSON). That means issues in the backend (like a pipe blocking) can take down the whole site.
Solution sounds simple: Split it up. Have a backend that runs the pipes, and a frontend that delivers the results. That way, the backend could be scaled up horizontally easily (no session issues there anymore!). And it also could make it easier to add something pipes is missing: a push infrastructure, so pipes can run for example when an input feed gets an update.
So I built this. The prototype was to have a
pipe_runner.rbscript that gets called by the frontend, which can run a pipe by just getting its json representation (which we have in the database anyway). A full system could then queue this or use it in whatever form ideal.The performance was too bad to continue. I seems the overhead of splitting frontend and backend - maybe the IPC, maybe missed issues with caching, I could not pinpoint it - made the whole system a lot slower. There was no way this could work on production.
Maybe the monolith is the right approach despite the issues described above. This re-architecture could also just need a different approach - maybe the backend needs to be a long-running server process again, maybe in combination with other performance improvements.
Just recording results, I will close this issue immediately as additional worked in that direction is not planned for now, though input is welcome.