Restructure data upload and filter

[mcwimm]

This PR includes the restructering of data upload and filter by time. @Ale0430 @cailadd90 please check if this works for you. Please also check if the time span shown in data > upload > table matches the time in the uploaded csv.

At the moment you should see

• error messages:
  • if the data upload did not work
  • data set contains less than two days of data
• warning messages:
  • data set contains less than 15 days
  • files are stored but no project is defined
  • if time windows of R and T do not overlap
  • if mean(Acc) > 0
  • raw data set has more than 2 columns
• information messages
  • files are stored in project directory
  • time windows overlap

Short description how you can see this branch on your computer:

1. Clean your working tree (nothing unstaged -> git status gives an overview
2. Fetch remote changes -> git fetch --all
3. Checkout this branch -> git checkout shiny_m
4. Leave a comment here :)
5. Switch back to your project -> git checkout your-branch

[Ale0430]

Warning message on "Data Upload".. with data frame is too small... change to " Data set too small to rely on result interpretation"

[cailadd90]

Cai WOW! Very impressed with the new upload and filter sections. I've experimented a little by uploading raw data from a B4+ (2nd) Mini Buoy. The timespan shown was correct. Looking really great.

Here are my suggestions:

EDIT: As per request, I've now added H, M, L to each comment according to priority.

General L • In the same way that the header remains fixed in place when you scroll up and down, can the menu on the left remain fixed as well? L• Can the font for project / target/ reference displayed at the top be white? And maybe have a matching font to the Mini Buoy App title? H• Maybe a good idea for all menu windows to have a 1 column tab along the top with a summary of what to do on each page?

Project settings H• Without more explanatory text, I think setting files / projects is a little confusing. I’m not entirely what the difference is between selecting a folder or setting a project. I wonder if it’s better that, everytime data is uploaded or downloaded, a browser window appears for the user to navigate to the right place, and so do away with using the project approach? For example, I worry that users will think it didn’t work when data was saved, as there was no prompt for where to save the data, or users might not realise /remember that new directories were created where the data / figures are saved. Moreover, would the user need to allow the App to modify directories on their computer? H• Current project and Current project directory + output [ character(0) ] display items aren’t helpful until rendered. Can the display be switched off until something is passed to them? L• You can create/set a project without first selecting a folder, giving the output ‘NULL’. Can this be stopped? One option might be to integrate folder select and project creation in one button. L• Without knowing what figure you’re going to title, it’s hard to decide what to enter here L• Visualisation window needs “Visualisation” as a title. When you minimize the window, nothing is shown. L• Is colour selection optional? Or maybe default option can be shown? L• It might be better to have the Visualisation window minimised as default, to keep the Project settings section minimal M Rewording • “Project settings” -> “Settings” (i.e., menu bar on the left) • “String added to file names” -> “Prefix for all files exported in this App” • “File name” -> “Prefix” • Figure theme -> remove “(ggplot)” • “Bw” -> “Black & white”

Data upload L• When we click “Browse…”, can it default to the project directory (working directory)? H• The summary dataset reloads every time a new “input file type” is selected. These shouldn’t be linked L• If the data can’t be read, it would be good to have a warning message like “Error: could not read file. Is the file in the correct format? Please refer to the Mini Buoy Handbook how to download data in the correct format for this App”, rather than the generic error being printed. L• Can we hide the column names, i.e. hide “Variable” and “Value”? Not helpful here. L• We don’t need date and time as separate columns, no? Just dateime and Acceleration? H• For the summary table, change the order/names slightly: Column names, Survey length (days), First date, last date, Mean acceleration (max and min), Median acceleration (1st and 3rd quantile), Number of recordings H• Can we round the acceleration data to 2 decimal places? L• Maybe a message after clicking “Use data” saying something like “Target data set”? M Rewording • “Upload data” -> “Upload” (i.e., menu bar on the left) • Titles can be simplified -> “Target” and “Reference” • Left panel: “Choose Target site CSV file” -> “Choose the target site data (.csv file)” • Right panel: “Choose Target site CSV file” -> “(optional) Choose the reference site data (.csv file)” • “Input file type” -> “Select Mini Buoy design used” • “Summary data set” -> “Summary of the data” • “datetime” -> “Date”

Filters M• Could start and end times default to the first and last entry for that date? Stops one trying to select a time earlier than the first record… H• The line and scatterplot still takes a while to render. Can you aggregate to 1/10 min windows before plotting? We don’t need to display all the data M Rewording • “Filters” -> “Filter” (i.e., menu bar on the left) • Titles can be simplified -> “Target” and “Reference” • “Remove NA-rows” -> “Remove blank rows • “Time filters” -> “Select the start and end dates/times of the survey” • “Raw data figures” -> “Check the raw acceleration data” • “View data set” -> “View data” • “Save csv” -> “Save filtered data”

Thanks!

[mcwimm]

@cailadd90 I updated the 'settings ' section -> see my comments below behind '->'

Project settings

• [x] H: optimize select folder + create project -> I updated the 'project' box. Please let me know what you think. All texts can be easliy replaced. I welcome suggestions :)
• [ ] L_ Without knowing what figure you’re going to title, it’s hard to decide what to enter here -> Here we can add a help icon later
• [x] L• Visualisation window needs “Visualisation” as a title. When you minimize the window, nothing is shown.
• [x] L• Is colour selection optional? Or maybe default option can be shown? -> Yes, it's optional. Added 'optional' to the title. Better?
• [x] L• It might be better to have the Visualisation window minimised as default, to keep the Project settings section minimal -> Good idea. Both boxes on the right are collapsed by default now as they are both optional
• [x] M Rewording • “Project settings” -> “Settings” (i.e., menu bar on the left) • “String added to file names” -> “Prefix for all files exported in this App” • “File name” -> “Prefix” • Figure theme -> remove “(ggplot)” • “Bw” -> “Black & white”

[Ale0430]

When going to shiny_m branch and running the app, several things look different from yesterday (i.e., Mini buoy models with old names (M"Mini Buoy Model 1", and "MB1"). I think I'm not being successful with the "git fetch --all" command

I'm working on the hydrodynamics functions script for now, so no need to answer for now.

[mcwimm]

@cailadd90 I redesigned the uplaod area. Now no data set is loaded by default but the user can still select to use a default data set. Otherwise, the default is always loaded first and the user has to wait until he/she can uplaod own data.

Data upload

• [ ] L• When we click “Browse…”, can it default to the project directory (working directory)? -> Not that I know
• [x] H• The summary dataset reloads every time a new “input file type” is selected. These shouldn’t be linked -> Can be solved as soon as we know how the raw csv files look like
• [x] Order: First select Type and brwose
• [x] L• If the data can’t be read, it would be good to have a warning message like “Error: could not read file. Is the file in the correct format? Please refer to the Mini Buoy Handbook how to download data in the correct format for this App”, rather than the generic error being printed. -> This will be implemented as soon as we have examples for all raw data types
• [x] L• Can we hide the column names, i.e. hide “Variable” and “Value”? Not helpful here. -> For now I overwrote them with empty strings. Not the best solution.
• [x] L• We don’t need date and time as separate columns, no? Just dateime and Acceleration? You're right, at the moment it is not used. I removed the columns
• [x] H• For the summary table, change the order/names slightly: Column names, Survey length (days), First date, last date, Mean acceleration (max and min), Median acceleration (1st and 3rd quantile), Number of recordings
• [x] H• Can we round the acceleration data to 2 decimal places?
• [x] L• Maybe a message after clicking “Use data” saying something like “Target data set”?
• [x] M Rewording • “Upload data” -> “Upload” (i.e., menu bar on the left) • Titles can be simplified -> “Target” and “Reference” • Left panel: “Choose Target site CSV file” -> “Choose the target site data (.csv file)” • Right panel: “Choose Target site CSV file” -> “(optional) Choose the reference site data (.csv file)” -> Add optional maybe to the box title • “Input file type” -> “Select Mini Buoy design used” • “Summary data set” -> “Summary of the data” • “datetime” -> “Date”

[Ale0430]

on computing hydrodynamics.

• [x] aggregate function is now used to calculate Median and quantile acceleration in time (minutes for B4, and B4+, 10 min for pendant
• [ ] Data frames on Target and Reference labeled as "Pendant" have loggings every minute -> Ceiling and floor functions are used to aggregate within the closest 10 min. mark
• [x] aggregate is also applied to compute mean flooding events per day.
• [ ] The aggregate() function slows down the hydrodynamics function but not the 'statistics' function.
• [ ] Will try to change aggregate for group_by() -> hope I can use it. - in the meantime, with patience aggregate() works fine

[mcwimm]

Filters

• [x] Error messages when no files are uploaded
• [x] M Could start and end times default to the first and last entry for that date? Stops one trying to select a time earlier than the first record…
• [x] H The line and scatterplot still takes a while to render. Can you aggregate to 1/10 min windows before plotting? We don’t need to display all the data
• [x] M Rewording • “Filters” -> “Filter” (i.e., menu bar on the left) • Titles can be simplified -> “Target” and “Reference” • “Remove NA-rows” -> “Remove blank rows • “Time filters” -> “Select the start and end dates/times of the survey” • “Raw data figures” -> “Check the raw acceleration data” • “View data set” -> “View data” • “Save csv” -> “Save filtered data”

[mcwimm]

For now, I would suggest to have only 1 default data set (with target and reference). So if the user choose to use 'default data' the mini buoy design is preselected and can't be changed. However, I don't know which mini buoy design of the default data set, so I assume it to be B4.

[mcwimm]

Date upload

New data upload is based on the files provided in TestSet/MB1 and Test data/Raw

Based on the files provided, data upload for B4 and B4+ is similar and as follows:

• all rows before '*DATA' are skipped
• columns 1 and 2 are assigned to be datetime and acceleration

this will show an error when uploading e.g. file L6.csv (provided in Raw)

Pendent data upload is different (raw file has > 2 columns but they are empty):

• columns 1 and 2 are assigned to be datetime and acceleration
• blank acceleration lines are removed
• if columns are completely empty they are removed and a warning is shown

ALL types need one column containing date-time

@cailadd90 @Ale0430 does this work for you?

[Ale0430]

fastPOSIXct () On my side testing origin/hydro file sI can upload data but when applying filters the app closes and an erro message shows on the console "could not find function "fastPOSIXct"

installing "fasttime" and loading the library solved this issue

Action

• [x] H add to set-up.R file under # Package to deal with time serien -> rollmean(), fastPOSIXct() if(!require("zoo")) install.packages("zoo") if(!require("fasttime")) install.packages("fasttime")

[Ale0430]

About page

Mini Buoy model images updated to diagram figures

Actions:

• [ ] L Cai -> higher resolution/quality images and decide whether we keep diagrams or photographs

[mcwimm]

fastPOSIXct () On my side testing origin/hydro file sI can upload data but when applying filters the app closes and an erro message shows on the console "could not find function "fastPOSIXct"

installing "fasttime" and loading the library solved this issue

Action

• [x] H add to set-up.R file under # Package to deal with time serien -> rollmean(), fastPOSIXct() if(!require("zoo")) install.packages("zoo") if(!require("fasttime")) install.packages("fasttime")

Added fasttime package to setup file, but in hydro branch - in this branch it was still activated