This repository contains the source code for the genomic incidence tracker for the Seattle Flu Project

Working prototype available here. Username: test, password: mockdata. Real results data are not yet available online.

Status: Working prototype

• Pathogen, map-detail, primary variable & group-by variable sidebar-selectors are working.
• Group-by variables trigger faceting, and one is able to toggle between faceted maps or tables.
• Tables can switch between displaying counts & percentages.
• Maps are simple shapes and only display 2-category chloropleths (i.e. boolean variables or variables with 2 categories).
• Hovering over map demes displays further info.
• Authentication (for login & API access)

Relevant links:

• GitHub issues for this project
• Simulated Data repo
• Flu Incidence (Observable Notebook)
• inVision prototype
• early map prototype
• geoJSON files used here
• auspice

Installing / Running

Make sure node & npm are avalable in your current environment. If you use conda, you can run:

conda env create -f environment.yml

Install dependencies etc:

npm install
source scripts/set_env_vars.sh ## only for local instances

Run:

npm run develop # development mode
npm run build && npm run view # production mode

Username: "local", password: "aaa" (local only)

Results & other files consumed by the client

Note that data formats will most probably change.

results.json

Due to current privacy concerns, no real results are committed to this repo. Mock data is available at "data/results.json" which is generated by "data/generate_mock_results.py". If "real" data is available at "./dataPrivate/results.json" (which is gitignored) then it is preferentially sourced by the server (make sure this data has the same shape). Here is an example of a data point (data not real):

[
  {
    "sex": "female",
    "residence_census_tract": "53033009300",
    "age": 39,
    "flu_shot": false,
    "pathogen": "h1n1pdm"
  }
]

Note that the pathogen key should be left out if the diagnosis is unknown.

data/variableChoices.json

contains a number of settings used. For instance, what sidebar options should there be, what labels should they have, what are the keys in the "results" file associated with these etc. It also contains the variable type and binning instructions for continuous types. The format of this file has not been thoughtfully considered, and should be revisited once we have finalised the form and contents of the results file.

Testing Server

This repo is running as a heroku server at https://genomic-incidence-tracker.herokuapp.com

Development Aims / Notes

These are based on experience (good & bad) with auspice over the past two years. This project will (hopefully) have contributions from a number of scientists and developers. This section aims to provide some guidelines and reasoning for directions taken.

React

We're using the latest stable version of React (16.8). This includes Hooks, which should remove the need to use classes for react components. As "there are no plans to remove classes from React", both can be used -- let's focus on clarity rather than the specific API used. This styleguide is a good read and has a number of useful and sensible suggestions.

Redux

It has been extremely helpful with Auspice to have a central source of truth. While there are plenty of "you don't need redux" blog posts, I think it's currently the best solution for an app like this. (While the new context API is useful for some things, it is too slow to store all the state for an app like this. See this thread for more details, including why it's still implemented as a HOC rather than a hook.) Note that state limited to a component can -- and should -- simply use this.setState and this.state (or the useState Hook).

In the future we may choose to store state at the root level, e.g. using a useReducer hook and passing dispatch down like this. But for now I think redux (and the provided dev tools) is the right choice.

Selectors

Selectors are a conduit to pass data between the reducers (redux store) and components. They allow the shape of data stored in the reducers to change without having to rewrite component logic. For instance:

// REDUCER
export const selectDemes = (state) => {
  // compute available demes from redux state
  return demes;
};
// COMPONENT
const mapStateToProps = (state, props) => {
  return {
    demes: selectDemes(state, props)
  };
};

Most of the time, this is the best place to transform data. If the transforms become expensive, these selectors can be memoised, meaning they do not rerun unless their arguments have changed. We use reselect for this.

Authentication

We are currently using basic authentication over HTTPS to log in. Currently there is only one calid username/password, stored as environment variables and accessed by the server (this is why it's necessary to run source scripts/set_env_vars.sh locally). The heroku server has different values here (which aren't public).

Authenticated users are supplied with a JWT which allows someone to remain logged in and authenticate API calls (TO DO).

Styled Components

We recently started using these in Auspice and have found them much superior to the previous situation which was a mixture of CSS, "global" styles imported into files, and component-specific inline styles. Similar to state management, there are plenty of articles about CSS-in-JS, both good and bad. Using styled components should facilitate consistent theming and provide a central place to change things as needed.

Geographic visualisation

Currently we only display simple shapes using d3 -- an appropriate stopping point for the April deadline.

Mapbox

We will (eventually) use mapbox to display the maps for this project. Libraries: mapbox-gl -- used in the map prototype repo, and react-mapbox-gl -- used in the seattle flu website. TO DO.

D3 - React

Ract hooks have vastly improved the react-d3 interface. We are currently writing all D3 (rendering) code for each chart inside a useEffect hook. See src/components/table/* and src/components/geo/* for examples.

Transitions:

The table shows (one possible way) to allow transitions. This is made slightly difficult as useEffect doesn't provide access to "previous props", however we can use a reference value (which avoids React rerendering) to store previous values as neccessary and then compute the appropriate update (rerender, transition, etc).

Potential bugs:

The interface between react & D3 has been a source of a number of auspice bugs -- for instance, if a react update wants to change both colour and shape, then running

selection.style("fill", (d) => d.fill);
selection.style("stroke", (d) => d.stroke);

can lead to unexpected results! (It's even worse if transitions are happening). By using a single imperitive D3 "update" call inside the useEffect hook this can be avoided.

Mobile

We should develop with both mobile & desktop views in mind, but de-emphasize interactions. For the April 2019 prototype mobile will not be considered.

Linting

Please use linting for all code -- the rules are intended to be helpful and prevent bugs as well as helping code remain consistent between developers (along with the editor config file). The rules aren't supposed to be a hinderence, so let's change them as needed. Feel free to deliberately disable for certain lines (// eslint-disable-line) as applicable.

Types

A lot of people seem to love TypeScript (JS + types), but I have no experience with it. If there are strong opinions then we could definately use it (my understanding is that you can use it on a file-by-file basis, but that may defeat the purpose).

Likewise, we used to use React's PropTypes in Auspice, but have drifted away from them for no real reason. I suggest we use them in this project, along with defaultProps and displayName where applicable.

Testing

Ideally, unlike auspice, we will have some tests in this project! These can probably wait until after we have a working prototype. Having only used them a little bit, I like jest and react-testing-library.

Travis-CI: TO DO.

Documentation in the code

Where possible, using simple JSDoc comments are a good idea!

Github

Non-time-critical changes should be merged into master using PRs with code review.

Data requests (server API)

The current implementation sources all API handlers from ./server/api.js. These handlers should be easily imported into another server -- e.g. the current seattleflu.org server -- as needed in the future. One example API is provided: localhost:4000/getData or https://genomic-incidence-tracker.herokuapp.com/getData

File Structure

Note that the data structures and APIs have not been finalised.

• ./data/* - Data files (geoJSON, mock-results etc)
• ./privateData/* - Local, non-committed, data files (this is gitignored).
• ./dist/* - Transpiled JS code. Gitignored.
• ./server/* - The CLI interface to running the app & helper files.
• ./src/* - All client javascript code
  • components/* - All React components
  • reducers/* - Redux reducers
  • actions/* - Redux actions
  • middleware/* - Redux middleware
  • styles/* - Themes etc (made available to components using styled-components)
  • utils/* - Misc functions etc

License and copyright

Copyright 2019 Bedford Lab

Source code to Nextstrain is made available under the terms of the GNU Affero General Public License (AGPL). Nextstrain is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details.