njeirath / brenda-web

Brenda-Web - A web based frontend for Brenda
http://www.brenda-web.com
Other
12 stars 13 forks source link

Cannot pull task details from custom AMI #88

Closed aliasguru closed 7 years ago

aliasguru commented 7 years ago

I've created a custom brenda AMI on AWS and am successfully rendering with it. I can use Brenda-web to submit jobs just fine. The only thing I don't know what to do about is an issue in the Dashboard: A few of the details don't update. These are:

When debugging the site, I noticed in the console that the communication with the AMI failed because of a CORS error. It basically tells me that "No Access-Control-Allow-Origin header is present on the requested resource. This happens when brenda-web tries to read the uptime.txt file from the server.

However, using one of the provided AMIs works just fine. So my question really is: What do I need to do to enable CORS communication in my custom AMI?

The exact error in Firefox (German I'm afraid): Quellübergreifende (Cross-Origin) Anfrage blockiert: Die Gleiche-Quelle-Regel verbietet das Lesen der externen Ressource auf http://54.236.31.153/uptime.txt?d=1508143629217. (Grund: CORS-Kopfzeile 'Access-Control-Allow-Origin' fehlt).

njeirath commented 7 years ago

@aliasguru Can you specify which version of Ubuntu your AMI was based off of?

The quick answer is to reference the very last section named "Nginx Path" in the Tutorial and validate that the "Nginx Path" field is being set appropriately on the "Setup Job" tab under the "Add Workers" section (this will depend on the version of Ubuntu you used).

Some more background on why the issue may be occurring:

CORS is a security mechanism implemented by most browsers to prevent web applications from making calls to servers on a different domain CORS details. This occurs in the dashboard because the domain of the site is brenda-web.com (or localhost if running locally) and this page is trying to access data on a different domain (54.236.31.153 in your case). To prevent this from being an error, HTTP provides for the Access-Control-Allow-Origin header which can be set to a particular domain on the server or simply '*' to allow all origins.

To serve up the data from each of the workers to drive the dashboard functionality we install a light weight web server Nginx. This is automatically installed by the startup script when brenda-web starts up an EC2 instance. We setup a couple cron jobs to push relevant files to the folder where nginx serves up files and also set the Access-Control-Allow-Origin header to '*' to allow all cross domain connections.

The full startup script can be seen on the brenda-web site under the "Setup Job" tab at the very bottom in the text area labeled "Startup Script", I've pulled out the relevant portion of the script below:

sudo apt-get -y install nginx
sudo sed -i '29 i\ add_header 'Access-Control-Allow-Origin' '*';' /etc/nginx/sites-enabled/default
sudo echo "* * * * * root tail -n1000 /mnt/brenda/log > /usr/share/nginx/html/log_tail.txt" >> /etc/crontab
sudo echo "* * * * * root cat /proc/uptime /proc/loadavg $B/task_count > /usr/share/nginx/html/uptime.txt" >> /etc/crontab
if ! [ -d "$B" ]; then
  for f in brenda.pid log task_count task_last DONE ; do
    ln -s "$B/$f" "/root/$f"
    sudo ln -s "$B/$f" "/usr/share/nginx/html/$f"
  done
fi
sudo service nginx restart

The high level steps in the script are to

  1. Install nginx
  2. Setup the Access-Control-Allow-Origin header in the nginx configuration
  3. Setup a few cron jobs to populate the directory from which nginx serves web pages
  4. Restart nginx

Since you're receiving the CORS error that suggests that something went wrong during the startup process and the appropriate headers weren't set in the nginx config file. If you're able to make it public, the AMI ID you're using would help so I can spin up a worker to see what might be going on. Otherwise any logs you're able to provide from the worker startup would be helpful or at the very least the version of Ubuntu you're using.

aliasguru commented 7 years ago

Hi njeirath,

Very helpful information, thank you! When I ssh into a running instance, I can verify that the /etc/nginx/sites-enabled/default does contain the Access-Control-Allow-Origin part of the setup. It's a bit funny, because that line is added twice for some reason. My assumption is that when I created the image, I did that from an already running brenda or brenda-web instance. Maybe this is what breaks things. Could it be that I need to actually uninstall* nginx in my case before creating the AMI?

I also see that the latter part of the script which is supposed to create the symlinks in the root account does not run. Neither in /root nor in /home/ubuntu, any symlinks are present. But they are present in the /usr/share/nginx/html folder. Now I'd love to provide logs to you. Which ones do you require, and how can I retrieve them? You're looking for dmesg or something different? The root files in nginx exist, and are populated as expected.

My details: I started with an Ubuntu 16.04 image, which I updated to the most recent available fixes using sudo apt upgrade. I also installed the AWS GPU driver, which is working fine on the G3 instances I'm using. I also installed a bunch of custom tools there, so for now I'd like to avoid making the AMI public. If it's the only way to resolve it, we can discuss it later again.

For now, I think cloud-init-output.log (you see the last 100 lines) might give the most information, as it states that apt did not install nginx, because the latest version is already there:

S . X == . o B #.= + .o @ Xo . . ooo.=.% .. oooo..=* +----[SHA256]-----+ Generating public/private ed25519 key pair. Your identification has been saved in /etc/ssh/ssh_host_ed25519_key. Your public key has been saved in /etc/ssh/ssh_host_ed25519_key.pub. The key fingerprint is: SHA256:+W1ghyePRgYUJl70GGPdt6C1XkeD77AzMO1U8ventYA root@ip-172-31-8-198 The key's randomart image is: +--[ED25519 256]--+ ..Xo . . . * =. +o.o. . o .o.+=o. o.+.+ooo S B.O.+.o = E.* .+ + + +oo . . ..

+----[SHA256]-----+ Cloud-init v. 0.7.9 running 'modules:config' at Wed, 18 Oct 2017 08:46:12 +0000. Up 37.90 seconds. Hit:1 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial InRelease Get:2 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial-updates InRelease [102 kB] Get:3 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial-backports InRelease [102 kB] Get:4 http://security.ubuntu.com/ubuntu xenial-security InRelease [102 kB] Get:5 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial-updates/main Sources [278 kB] Ign:6 http://ppa.launchpad.net/irie/blender/ubuntu xenial InRelease Get:7 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial-updates/multiverse Sources [7,208 B] Get:8 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages [642 kB] Get:9 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial-updates/main Translation-en [269 kB] Get:10 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages [540 kB] Get:11 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial-updates/universe Translation-en [220 kB] Get:12 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial-updates/multiverse amd64 Packages [15.3 kB] Get:13 http://security.ubuntu.com/ubuntu xenial-security/main Sources [97.9 kB] Get:14 http://security.ubuntu.com/ubuntu xenial-security/multiverse Sources [1,140 B] Get:15 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages [370 kB] Ign:16 http://ppa.launchpad.net/irie/blender/ubuntu xenial Release Get:17 http://security.ubuntu.com/ubuntu xenial-security/main Translation-en [164 kB] Get:18 http://security.ubuntu.com/ubuntu xenial-security/universe amd64 Packages [175 kB] Get:19 http://security.ubuntu.com/ubuntu xenial-security/universe Translation-en [93.0 kB] Get:20 http://security.ubuntu.com/ubuntu xenial-security/multiverse amd64 Packages [2,756 B] Ign:21 http://ppa.launchpad.net/irie/blender/ubuntu xenial/main amd64 Packages Ign:22 http://ppa.launchpad.net/irie/blender/ubuntu xenial/main all Packages Ign:23 http://ppa.launchpad.net/irie/blender/ubuntu xenial/main Translation-en_US Ign:24 http://ppa.launchpad.net/irie/blender/ubuntu xenial/main Translation-en Ign:21 http://ppa.launchpad.net/irie/blender/ubuntu xenial/main amd64 Packages Ign:22 http://ppa.launchpad.net/irie/blender/ubuntu xenial/main all Packages Ign:23 http://ppa.launchpad.net/irie/blender/ubuntu xenial/main Translation-en_US Ign:24 http://ppa.launchpad.net/irie/blender/ubuntu xenial/main Translation-en Ign:21 http://ppa.launchpad.net/irie/blender/ubuntu xenial/main amd64 Packages Ign:22 http://ppa.launchpad.net/irie/blender/ubuntu xenial/main all Packages Ign:23 http://ppa.launchpad.net/irie/blender/ubuntu xenial/main Translation-en_US Ign:24 http://ppa.launchpad.net/irie/blender/ubuntu xenial/main Translation-en Ign:21 http://ppa.launchpad.net/irie/blender/ubuntu xenial/main amd64 Packages Ign:22 http://ppa.launchpad.net/irie/blender/ubuntu xenial/main all Packages Ign:23 http://ppa.launchpad.net/irie/blender/ubuntu xenial/main Translation-en_US Ign:24 http://ppa.launchpad.net/irie/blender/ubuntu xenial/main Translation-en Ign:21 http://ppa.launchpad.net/irie/blender/ubuntu xenial/main amd64 Packages Ign:22 http://ppa.launchpad.net/irie/blender/ubuntu xenial/main all Packages Ign:23 http://ppa.launchpad.net/irie/blender/ubuntu xenial/main Translation-en_US Ign:24 http://ppa.launchpad.net/irie/blender/ubuntu xenial/main Translation-en Err:21 http://ppa.launchpad.net/irie/blender/ubuntu xenial/main amd64 Packages 404 Not Found Ign:22 http://ppa.launchpad.net/irie/blender/ubuntu xenial/main all Packages Ign:23 http://ppa.launchpad.net/irie/blender/ubuntu xenial/main Translation-en_US Ign:24 http://ppa.launchpad.net/irie/blender/ubuntu xenial/main Translation-en Fetched 3,182 kB in 10s (292 kB/s) Reading package lists... W: The repository 'http://ppa.launchpad.net/irie/blender/ubuntu xenial Release' does not have a Release file. E: Failed to fetch http://ppa.launchpad.net/irie/blender/ubuntu/dists/xenial/main/binary-amd64/Packages 404 Not Found E: Some index files failed to download. They have been ignored, or old ones used instead. Reading package lists... Building dependency tree... Reading state information... nginx is already the newest version (1.10.3-0ubuntu0.16.04.2). The following packages were automatically installed and are no longer required: adwaita-icon-theme at-spi2-core colord colord-data dconf-gsettings-backend dconf-service dkms glib-networking glib-networking-common glib-networking-services gsettings-desktop-schemas humanity-icon-theme lib32gcc1 libatk-bridge2.0-0 libatspi2.0-0 libc6-i386 libcairo-gobject2 libcapnp-0.5.3 libcolord2 libcolorhug2 libcuda1-304 libdconf1 libegl1-mesa libepoxy0 libexif12 libfontenc1 libgbm1 libgphoto2-6 libgphoto2-l10n libgphoto2-port12 libgtk-3-0 libgtk-3-bin libgtk-3-common libgudev-1.0-0 libgusb2 libieee1284-3 libjansson4 libjson-glib-1.0-0 libjson-glib-1.0-common libltdl7 libmirclient9 libmircommon7 libmircore1 libmirprotobuf3 libprotobuf-lite9v5 libproxy1v5 librest-0.7-0 libsane libsane-common libsoup-gnome2.4-1 libsoup2.4-1 libwayland-client0 libwayland-cursor0 libwayland-egl1-mesa libwayland-server0 libxaw7 libxfont1 libxkbcommon0 libxkbfile1 libxnvctrl0 libxtst6 libxvmc1 ocl-icd-libopencl1 policykit-1-gnome python3-xkit screen-resolution-extra ubuntu-mono x11-xkb-utils xfonts-base xfonts-encodings xfonts-utils xserver-common xserver-xorg-core xserver-xorg-legacy Use 'sudo apt autoremove' to remove them. 0 upgraded, 0 newly installed, 0 to remove and 4 not upgraded. Cloud-init v. 0.7.9 running 'modules:final' at Wed, 18 Oct 2017 08:46:14 +0000. Up 40.43 seconds. Cloud-init v. 0.7.9 finished at Wed, 18 Oct 2017 08:46:32 +0000. Datasource DataSourceEc2. Up 57.20 seconds

njeirath commented 7 years ago

@aliasguru One thing you can try as a quick test would be to start up a worker, SSH into it and remove one of the "Access-Control-Allow-Origin" lines from the nginx config and restart the service and see if the dashboard starts working (might take ~30 seconds to a minute after the restart). If that works it would suggest nginx can't handle having 2 of those lines in the config and probably uninstalling nginx and rebuilding the AMI would probably be the easiest approach.

If that does in fact end up being the issue let me know and I can put in a check to see if the header is already set in the config file before adding it again.

aliasguru commented 7 years ago

@njeirath Have tried above suggestions, but no luck. The instance would still not allow any CORS communication after editing the default file and restarting nginx service. I then ssh'd into it again, purged nginx and created another custom AMI. When spinning that one up, nginx is installed via the script just fine, the "Access-Control-Allow-Origin" is now only present once. Still: No CORS communication possible. There must be something else.

I think as a next step I might just go ahead and reinstall everything on a new instance, which I run from your existing 2.79 AMI. Maybe then I have my tools working AND communication happening with the instance. Still wonder what the issue could be.

aliasguru commented 7 years ago

I think I found a little difference between the /etc/nginx/sites-enabled/default files on our AMIs. Yours reads:

server {
    listen 80 default_server;
    listen [::]:80 default_server ipv6only=on;

    root /usr/share/nginx/html;
    index index.html index.htm;

    # Make site accessible from http://localhost/
    server_name localhost;
 add_header Access-Control-Allow-Origin *;

    location / {
        # First attempt to serve request as file, then
        # as directory, then fall back to displaying a 404.
        try_files $uri $uri/ =404;
        # Uncomment to enable naxsi on this location
        # include /etc/nginx/naxsi.rules
    }

mine reads:

server {
    listen 80 default_server;
    listen [::]:80 default_server;

    # SSL configuration
    #
    # listen 443 ssl default_server;
    # listen [::]:443 ssl default_server;
    #
    # Note: You should disable gzip for SSL traffic.
    # See: https://bugs.debian.org/773332
    #
    # Read up on ssl_ciphers to ensure a secure configuration.
 add_header Access-Control-Allow-Origin *;
 add_header Access-Control-Allow-Origin *;
    # See: https://bugs.debian.org/765782
    #
    # Self signed certs generated by the ssl-cert package
    # Don't use them in a production server!
    #
    # include snippets/snakeoil.conf;

    root /var/www/html;

    # Add index.php to the list if you are using PHP
    index index.html index.htm index.nginx-debian.html;

    server_name _;

    location / {
        # First attempt to serve request as file, then
        # as directory, then fall back to displaying a 404.
        try_files $uri $uri/ =404;
    }

So it seems like there are different versions of nginx or at least the configuration file in use. Their order and content is quite different. I'm clueless as of how that's possible.

aliasguru commented 7 years ago

SUCCESS! I've ssh'd into another running instance now, and manually edited the /etc/nginx/sites-enabled/default file, entering the exact details which you have in yours. Once done and restarted nginx, the instance shares all details!

aliasguru commented 7 years ago

After further examination, two conditions need to be met:

1) the root path must match the one given by brenda-web on task submission 2) there must not be a double Access-Control-Allow-Origin line

Only if both conditions are met, the machines talk to each other.

njeirath commented 7 years ago

@aliasguru Great, glad you got it working and thanks for the explanation of what you found. I may try to build a replacement for nginx as that's the second time a version difference has bit me. Should hopefully help alleviate some of these issues in the future.