Brainstorm: Example/hypothetical project that achieves learning outcome

[lwjohnst86]

This space is for brainstorming ideas for what type of project/hypothetical example the participants could complete to show that they've achieved the learning outcome (e.g. kinda like the team project in the intro R course). They don't have to actually do it, but we should have an idea of what it would look like for them to have achieved the learning outcome.

[danielibsen]

Idea: each team presents on an analysis pipeline in a specific server environment. They should answer some questions and present how they handled their server.

The overall task will be the same. But the environment would be different.

[danielibsen]

List of types of environments:

• Danmarks Statistics
• VDI (public health)
• UK Biobank Research Analysis Platform
• Computer roam
• BIANCA (sensitive data server env in Sweden)
• only terminal access servers (GenomeDK)

[lwjohnst86]

• Given a type of server and what they could do for that server?
  • e.g. UKBiobank vs Public Health VDI vs DST vs terminal-access only
  • e.g. no access to internet or having a short time out (before server gets turned off) or with shared memory etc
• Follow guidelines on extracting results
  • e.g. type of plots, not saving data in Git, getting table results out, like we have with DST
• Project setup for some types of servers
  • e.g. like installing everything from the beginning etc
• Finding where the data are stored and using proper file path connecting the data
  • Processing script in data-raw to take larger data and select only data you need and save to project folder (git ignore)
  • Converting into parquet format? Saving in chunks
• Setting up renv? To deal with package dependencies
• Interacting with clusters? Distributed computing, so a task that requires parallel processing (for converting to the parquet format)
• Then a simple analysis of the large dataset, saving intermediate results
  • Especially if server times out after 24 hrs or whatever time
• Basic report with figures and tables that follow privacy guidelines

[lwjohnst86]

Sequence of steps for project:

• Starting point:
  • Learning how to identify what file storage format (e.g. csv or SAS dataset) there are and knowing how convert those files into more efficient formats (like Parquet or a SQL database)
  • Give them a few server environment types, and the same data but with different starting formats. And then they figure out the next steps based on that information
  • Multiple data is big enough to prevent doing it normal way (1 Gb or larger?)
• Explaining why the original data format might not be ideal and then converting the data into more efficient format
• Identify what the desired sample is for the dataset, only select and filter data they need for analysis
• Split the data into smaller chunk to prototype code (running code on all the data later)
• Run basic analysis (descriptive statistics)... Not modeling
• Implement some code to run with parallel processing
• Identifying which format data or items can be downloaded, and converting that to that format

Assumptions:

• Assume they have taken the intermediate course (need to know functionals and function-based workflows), and either have read or taken the advanced course or are familiar enough with targets