Demonstrate pod-dashboard on docker-compose and cloud.gov

[pburkholder]

https://github.com/GSA/project-open-data-dashboard/pull/188 is the initial PR for demonstrating docker-compose and use of CircleCI for smoke tests

[pburkholder]

@adborden and I reviewed the docker-compose PR and will incorporate --fail on curl, and test again end-to-end.

There are a few steps from the earlier issue to account for:

• [x] Stream log_message messages to stdout or stderr so cloud foundry logging can capture
• [ ] Fix redirect issues for localhost:3000
• [x] Assess circleci smoke tests. We probably want separate test to a) to mimic the VCAP_SERVICE env settings, and b) simple .env (ansible-style).
• [x] Ensure backwards compatibility with Ansible and EC2 until we can migrate.
• [ ] Test S3 by setting max_-related filesizes to arbitrarily low limits, and then assuring things work on the S3 logic path.
• [x] - Ensure steps in the ansible playbooks, rooted at https://github.com/GSA/datagov-deploy/blob/develop/ansible/dashboard-web.yml, are accounted for
• [ ] - Remove sample php files and references to env -- let's just use development to ensure adequate logging

[pburkholder]

Adding a note here on streaming logs, which I'll come back to after making sure the docker-compose branch works in the CI environment (w/ Ansible).

Most PHP web apps will treat STDOUT as what gets passed to the web server and back to the client, which is why the log_message and write_log functions of CodeIgniter write to a specific path in application/logs. We should be able to use the fpm.d tricks at https://github.com/cloudfoundry/php-buildpack/issues/256 and/or https://github.com/cloudfoundry/php-buildpack/issues/202 to set FPM_CATCH_WORKERS_OUTPUT to yes.

We'll also need to extend CI_Log class with MY_Log and override write_log with our function that looks for $config['log_stdout'] to write to STDOUT.

[pburkholder]

I have ansible and test-kitchen working locally and in the CI environment, so I'm trying to reconcile how we configure dashboard between bsp/ansible and docker and cloud.gov. The WIPI branch at docker-compose-no-templating moves all the application variables to env_vars. The application/config/*.php all are pulled from datagov-deploy templates dir, and then use env vars with the expectation that:

• bsp will use ansible to write a .env file that is then picked up
• docker-compose will write a .env file (or just use the ENV)
• cloud-gov bootstrap.sh will likewise write an env file.

Tuesday I should be able to test and then we can get the BSP env running on the new branch.

[pburkholder]

Some observations about use of S3:

S3 is used in a few places:

• for put/fetch of csv_to_json
• for archiving data.json and digitalstrategy

The archive_file function will use config[use_local_storage] anytime it's called but the logic doesn't apply when to datajson_lines. The function has a specific exception for that.

S3 also used in the office_detail view and the office controller to build URLS that reference the objects in the archive.

This will require 2 buckets, since they have different ACLs attached.

Also, not all uploads use S3, or all use of archive, when use_local_storage is false. S3 is only used in these particular cases. So json validation uses upload but doesn't pay attention to use_local setting, for example

About some of the files:

• campaign.php
  • public function csv_to_json($schema = null)
  • once to PUT the file
  • again to GET the file
  • There is no remove from S3
  • We archive digitalstrategy.json and data.json using archive_file which puts a fetch date in the URL.
• campaign_model has
  • archive_file which calls `archive_to_s3
  • archive_to_s3 which calls put_to_s3 and stores with a PUBLIC acl
  • put_to_s3 which stores private by default
• views/office_detail
  • Builds a URL based on values of config/s3_bucket

[pburkholder]

I'm opening a PR since this is a good merge point before I start getting my hands in the PHP code. I've made a draft PR and need to fix circleCI and some other wonkiness, then I'll let you know when I'm ready.

I'm going to flesh out the what I've done and what needs to be done, then use my remaining hours this week, Monday and 1/2 of Tuesday to actually do those things.

Also, I'm pulling together a few notes for our conversation with Aidan tomorrow.

[pburkholder]

Current considerations for migrating to cloud.gov:

• Logging

  • Currently the only logs for CodeIngiter apps in BSP are the Apache/Nginx logs. Logs from the apps themselves, from log_message are not kept as config[log_threshold] is set to 0 (zero) in BSP. This means you don't get any code-level errors, like:

      ERROR - 2019-10-01 12:21:29 --> Severity: Warning  --> fgetcsv() expects parameter 1 to be resource, boolean given /home/vcap/app/htdocs/application/helpers/csv_helper.php 44
      ERROR - 2019-10-01 12:21:29 --> Severity: Warning  --> Invalid argument supplied for foreach() /home/vcap/app/htdocs/application/helpers/csv_helper.php 48
      ERROR - 2019-10-01 12:21:29 --> Severity: Notice  --> Undefined variable: row_new /home/vcap/app/htdocs/application/helpers/csv_helper.php 52

  • Most PHP web apps will treat STDOUT as what gets passed to the web server and back to the client, which is why the log_message and write_log functions of CodeIgniter write to a specific path in application/logs. If we want these logs:
  • We'd need to write our classes for log_message to write to STDOUT, then
  • We should be able to use the fpm.d tricks at cloudfoundry/php-buildpack#256 and/or cloudfoundry/php-buildpack#202 to set FPM_CATCH_WORKERS_OUTPUT to yes.

• S3 updates to use authenticators

  • The BSP code only need to reference a bucket and, thanks to IAM roles, could write to that bucket without needing to set credentials.
  • Credentials can be passed to the client by:
  • Using the environment variables , e.g.:
    • https://docs.aws.amazon.com/sdk-for-php/v3/developer-guide/getting-started_basic-usage.html
    • https://docs.aws.amazon.com/sdk-for-php/v3/developer-guide/guide_credentials_environment.html

  • Explicitly setting credentials, e.g.:

    $credentials = new Aws\Credentials\Credentials('key', 'secret');
    
    $s3 = new Aws\S3\S3Client([
      'version'     => 'latest',
      'region'      => 'us-west-2',
      'credentials' => $credentials
    ]);

• S3 Sync all archives https://s3.amazonaws.com/bsp-ocsit-dev-east-appdata/datagov/dashboard/archive/* to the S3 bucket for cloud.gov. Resources:

  • Using S3 credentials in cloud.gov
  • Using aws s3 sync

• Set up cron jobs.

  • There are two cronjobs:
  • index.php campaign status cfo-act download (takes about 5m)
  • index.php campaign status cfo-act full-scan (takes a lot longer)
  • Cloud.gov doesn't have built-in support for scheduling, so we can pursue a couple of other options for running the daily jobs.
  • One option -- run a worker process that mostly sleeps, but checks every 30m if it's "trigger time" to run the daily tasks, as in this example from cf-ex-drupal-8.
  • Whether it's triggered could be determined by whether a cronish table in the database has not had any updates in 24 hours.
  • For demo purposes, this example has a cronish app that just does the download job and then sleeps for 23h.
  • This worker process keeps running regardless of the stability of the web process, but uses the same droplet build.
  • Cloud.gov also doesn't support sending mail from the shell, which had been the method for sharing crawl jobs on BSP, e.g: php index.php crawl | tee /var/log/dashboard-cron.log | mail -s 'LabsData - Dashboard' default_email
  • The worker pattern for the crawl labels all crawl logs with a worker id, e.g.: [APP/PROC/WORKER/0] OUT Attempting to request http://www.sba.gov/data.json

• Run the dashboard app under /dashboard, by launching with cf push --route-path /dashboard

  • There may be issues with link writing and the value of config["base_url"]. It may be sufficient to unset config["base_url] and let CodeIgniter determine URLs internally, and have CloudFoundry rewrite for you. Or, you may want to emulate what's in the Ansible code with:

    if (isset($_SERVER['REQUEST_URI']) && 0 === stripos($_SERVER['REQUEST_URI'], '/dashboard')){
      $config['base_url'] .= '/dashboard';
      $cookie_path_prefix = 'dashboard';
    }

• Write your SSP ;)

• Proxy from https://labs.digital.gov/dashboard to https://(cloud.gov host)/dashboard

• Tear down existing dashboard hosts

[pburkholder]

I'm updating the title of this so we can close it

[mogul]

All remaining work identified here is now in other issues:

• Run cronjobs
• Log CodeIgniter errors
• Respect route path
• Use injected S3 config
• Alert when crawls fail
• Get ATO
• Migrate production service

[mogul]

The dashboard app is demoable on cloud.gov at https://app-rested-genet.app.cloud.gov/offices/qa