Beiwe single server AMIs are substantially out of date

[biblicabeebli]

The current AMIs listed in the 'Quick setup (single-server AMI)' section on the deployment instructions are out of date and running the Python 2 version of the codebase.

Updating an existing AMI is complex enough that we are not currently planning to provide a guide, we will have to create new AMIs.

The easiest way to upgrade existing systems is probably to do a database export or transfer, and then set up and run migrations on a new Python 3 AMI.

Please comment below if you are using the AMI and require assistance, so far I am only aware of one user who has been in contact via the forums.

[biblicabeebli]

The new AMI should use the new Python 3 Celery data processing architecture. the ami_cron_target should be removed from the repo.

[adityakamble49]

I have deployed Beiwe_v0.2 (ami-d97ddea6) AMI on my AWS EC2 and updated code as per given instruction here.

I have also updated python to 3.6 using Virtualenv and installed appropriate mod_wsgi for that python version. But I am facing URL Not found error when try to reach the server from my domain name. It was working earlier with old version.

I am able to run flask server in debug mode from ec2 and able to access from my EC2 which confirms code update is correct but having some issues with apache2 bindings with wsgi and flask routing.

Can anyone share their single server AMI apache2 configuration for latest AMI which will be useful for fixing the routing and URL not found error

[biblicabeebli]

hello adityakamble49! Thank you for your

I think the relevant fact for your endeavor is that mod_wsgi3 (or whatever the apache pluginy thing for python 3 is called) was never developed with anything other than the system python version in mind. You will need to install python36 (I think that is the package name under ubuntu 16.04? I think python34 and python36 are different packages.) and swap all usages of python over to python36. Or something like that.

Unfortunately, due to the structure of the new feature currently in active development (push notifications) the current AMI is going to go from seriously out of date to PROFOUNDLY out of date. It needs a more involved update.

I will try and bump priority of this soon. The best solution is for me (or ...you? :D ) to update the code in the launch script. There is an unexposed function do_create_single_server_ami that is either unfinished or out of date. (@zagorsky Josh, could you respond on whether it was finished and/or out of date?)

If anyone wants to take a whack they should go off the eli/push-notifications branch as it is where the most current feature dev is occurring.

The big change is that the AMI does not have celery configured. The old cron file needs to be replaced with the manager cron file. Haven't looked at it, other updates probably need to occur too. Like the python3 change for worker/manager servers everything python needs to become python3. Also it might be expecting ubuntu 16.04 and everything else has been update to 18.04.

[zagorsky]

There is an unexposed function do_create_single_server_ami that is either unfinished or out of date. (@zagorsky Josh, could you respond on whether it was finished and/or out of date?)

The function do_create_single_server_ami() in cluster_management/launch_script.py was finished but it may be out of date now; I haven't run and tested it in a few months.

[adityakamble49]

Thanks @biblicabeebli & @zagorsky for the updates.

Actually I am able to install Python 3.6 and configured it using Virtual Environment on AMI. In addition, I have compiled mod_wsgi [Reference] for Python 3.6 Specifically as AMI Ubuntu is only compatible with Python 3.5 and if we install mod wsgi with default apt, it will install it for Python 3.5. So the issue of Python Versioning is resolved.

This is how I am able to run the website in debug mode on port 8080.

I will check out do_create_single_server_ami() method for AMI Update

I have following concerns now.

• Do I need to update Apache 2 configuration files to support the latest code?
• As I already updated the code in AMI, If I follow these Instructions to update CRON jobs, will it configure celery?

Also Thanks for quick response guys.

Currently I am working on this to present demo in our class at UNC Charlotte. I am always passionate about Open Source and Contributing back to community. It would be great if you could point me to some issues added to GitHub which needed to be fixed. Happy to help !!

Thanks

[biblicabeebli]

Oh, well that's pretty nice I didn't know that was possible.

cron - yeah, I think those directions are correct. Just make sure the python executable in the cron task is the correct one. (pyenv which python may be helpful.)

Apache - config - not really, except that you will need to set up some ssl certificate as the app refuses non-https connections because this app deals with Personally Identifiable Information.

Right now you are okay if you are on the master or production branch. The cron job on the AMI uses our older method for running the data processing, it is just inefficient and has some obnoxious failure modes. The newer code uses celery to run some way better multiprocessing (and multi-server) data processing code. The code in development is even better, and it will be required for the new push notification based code update that should improve data collection on the app, which is not strictly necessary. (Status: it is written for Android, iOS is in development, and I'm setting up to test the full stack server + Android right now.)

You are good as soon as you can download data. Make sure your studies are running in test data mode so that you can get your raw data.

There are many small and large issues on the github issues page. Any input on issues labelled as discussion is desireable and does not require you write code, help wanted is pretty self explanatory, the enhancement issues might contain some low-hanging fruit.

good luck with the presentation!

[biblicabeebli]

Onnela Lab decided to stop supporting the AMI some time in the past year because the AMIs that did exist became so old that AWS stopped supporting them.

In principle it should not be difficult to adapt the deployment script to connect to an arbitrary server. Code burden is on adapting the existing single-server-setup code to whatever changes might be required to get it running.

Recommendation is to run with apache forwarding to UWSGI, that's what we do on the AWS Elastic Beanstalk servers (but those run AWS Linux, not Ubuntu).

[biblicabeebli]

We don't plan to pick this up without external and active requests, so I'm closing this issue. Feel free to comment, make pull requests, open new issues on this topic if you are interested.