search

algorithmic-music-exploration / amen-server

Server to run Amen analysis and return JSON results.
BSD 2-Clause "Simplified" License
7 stars 2 forks source link

Architecture #3

Closed tkell closed 7 years ago

tkell commented 7 years ago

Less sure about this, but I think that the server takes the audio, puts it on a queue, and then the queue (or two queues?) handle analysis and uploading the audio to S3.

It may also be possible to use some S3 magic here, and post the audio directly to S3 - maybe even use their Lambda stuff?

alnesbit commented 7 years ago

the queue (or two queues?) handle analysis and uploading the audio to S3.

Could you expand on this a bit more? For example, why two queues? In my mind's eye, and at my current level of understanding, this part is the core of the server.

It may also be possible to use some S3 magic here, and post the audio directly to S3 - maybe even use their Lambda stuff?

I agree that it would be interesting magic. But what would be real advantage of doing it this way? My concern is that it could unnecessarily tie the project deeply into S3, potentially making it difficult to port the system to a different "block storage" backing platform if that were so desired. Nevertheless, as I stated in another comment, I'm not necessarily opposed to S3, but rather, trying to understand our connection with it better.

tkell commented 7 years ago

I think I am wrong on two queues, yeah. In my mind, one queue would upload the audio to S3, and another would analyze audio on S3. The first queue is probably the server itself, as you say.

Assuming that AWS Lambda can do what we want (@psobot has said that it does not give you a lot of CPU power), it would be nice to not have to maintain servers and maintain fancy queues. I agree that the lockin to AWS is not great, especially if we want to put this forward as a "you can run your own server!" sort of thing.

tkell commented 7 years ago

Did some more reading about Lambda - to make a complex virtualenv run, it looks like you need to send it a zip file of that virtualenv. And, if you want to send C extensions, you need to compile them on an EC2 machine?

Given that we tell people to install Conda, this is looking less like it is for us.

tkell commented 7 years ago

Made a start of a thing that works locally at https://github.com/algorithmic-music-exploration/amen-server/tree/simple-tornado-server.

tkell commented 7 years ago

^^ moved the above to master