Storage of large data

[leewujung]

• [x] Allow storage of large amount of data to an external drive
• [x] Provide a strategy that works for a 3-day unattended recording
  • This is a pretty stringent requirement, so make sure you have a strategy for figuring out bugs if recording fails
• [x] Identify an external drive for purchase for field work in August

[aidanjohnson]

External Drives

Background

Mostly for initial research purposes, I have found these miscellaneous documents to be 'inspirational' guidelines for selecting an SSD:

• AMAR G3 Autonomous Acoustic Recorder by Jasco: While these are for marine mammals, the same general design specifications apply.

• AURAL M2 by Multi Electronique: Note that a setup guide on Multi Electronique shows a WD Blue SATA HDD. This recorder brings up a tradeoff and constraint to consider when choosing an SSD:

  1. The capacity of the drive is proportional to energy consumption.
  2. The data transfer rate must be sufficiently fast for the AIFF file creation.

• Comparison of autonomous passive recorders: This led me to the two recorders above in addition to OceanBase LADIN.

Desired Characteristics

• At least 1 TB of data storage capacity.
  • Tradeoff: cost with capacity.
• Energy efficient such that it can be powered by the DT7816, which requires 5 Vdc from an external source—i.e. a battery.
  • Tradeoff: capacity with energy consumption.
  • Type of battery: tradeoff of cost with energy capacity (more on this later—lead acid, alkaline, Li-ion?)
• Data transfer rate: Limited to speed of USB 2.0 (480 Mbps). Note that USB 3.0 is backwards compatible with USB 2.0 interfaces but is limited to USB 2.0 speeds (480 Mbps).
  • Limited to using USB and not SATA, and its derivatives, directly unless we use a converter/adapter cable (must be powered for >3.5 inch SSD and HDD, otherwise USB provides 5 Vdc sufficiently).
  • Limited to using an SSD that has a SATA interface (M.2 include) except for PCIe (NVMe).

Possible Candidates

For reference I compiled this spreadsheet for comparing a set of possible SSDs. The metric I used to compare (minimise) was the power cost per unit storage (mW per GB)–this is equivalent to maximising the storage benefit per unit power (GB per mW).

Conclusions

Considering the above analysis, I recommend the Samsung 860 EVO M.2 SATA 1 TB SSD (or even the 2 TB). In conjunction with the 860 EVO we would need to purchase an adapter cable for converting M.2 SATA to USB. This adapter and case from Amazon (or Newegg, etc) will do.

Since the DT7816 only has USB 2.0 instead of USB 3.0, we will be forced to live with 40 MB/s bandwidth instead of 400 MB/s. Although, there is a benefit of this constraint as the 2nd generation consumes less power compared to the 3rd generation (2.5 W vs. 4.5 W).

The following is a summary for estimated cost of these two components (including tax and shipping):

1. Samsung 860 EVO M.2 SATA 1 TB SSD—$262 (at most $300)
2. ELUTENG M.2 Enclosure USB3.0 to SATA—$11

@leewujung, let me know what you think.

[aidanjohnson]

Strategy

Goal

A reliable 3-day unattended recording regime.

Considerations

• Recording daily duty cycle
• Battery constraints
• Storage constraints
• Field ease-of-use

Duty Cycle

Cycle on and off recording for sunset and sunrise (the daily duty cycle), respectively, should maximise useful recording information while minimising energy consumption (i.e., runtime) and cumulative data size. Sunset and sunrise times, which are calculated for longitude and latitude coordinates, can be buffered with a margin of safety (already implemented in software). A satisfactory safety margin—1800 s, 3600 s, etc. before sunset and after sunrise—will need to be selected.

Battery

For how long can the chosen battery power a running board that alternates between idle and recording? The battery must be selected such that it can power the board for at least 3 days, in order to satisfy most practical purposes. Additionally the type of battery must be chosen. Is it best to use a Li-ion, alkaline, or lead acid battery?

Storage

See earlier discussion (above) on an external drive for the constraints faced. The energy consumption of the external drive must be factored into the minimum battery requirement calculation. We want to minimise the power cost per unit storage.

User Interface (UI)

The 8 LEDs on the board currently indicate if the given analog input channels are in operation. All turn off if no channels are active, i.e. the board is in idle and functioning normally or recording has failed. We will know if recording has failed if the current time falls within the on cycle and all channels are inactive. The use of these LEDs exhausts the built-in indicator functionality of the board. All additional indicators will less directly observable and therefore more difficult to use in the field.

The current version of the software prints to the terminal each time a recording file is written to storage. This can be observed on a user laptop when the program is first run via the serial console. Any errors, fatal or otherwise, are printed to the terminal and the program shuts down. The only way to tell would be to connect to the board via USB-Serial or to notice the analog input channel LEDs are off during the on cycle (of course this observable state could mean other errors such as incorrect current local time or other improper duty cycle behaviour). Lastly, it is unclear whether connecting to the board after the program has been running will interrupt the program and reboot the board or will let the user view the board's current state. My suspicion is the latter. This will need to be verified once @funnyabc is finished with the CUnit tests and the board (and USB-Serial cable) is back in my possession.

Regime

1. Connect the board to the field laptop via USB-Serial and Ethernet once the board has been supplied power, connected to external storage, and positioned (with microphones) where desired. This will be the field user's opportunity to synchronise the local time clock and run the software recorder. The write path to the external storage will need to be entered into the C source code and compiled beforehand. The external drive should have the path /media/....

2. The safety margin is to be specified in the command line when running the program. It can also be pre-defined and compiled beforehand.

3. Depending on the current local time, the board will either enter its idle or recording state. In the recording state the user should see the same number of LEDs flashing on as analog input channels selected. If the board is idling no LEDs should be lit. At this point the board can be left unattended.

4. The board will follow this finite-state machine.

5. For the 3-day period, the board will continue its duty cycle as specified. If the duration was set via command line when the recording was begun, the program will stop (i.e., return successfully) when complete. Otherwise, the program will need to be manually aborted via the serial console. Setting a recording duration will make the regime truly unattended as the program will stop when complete without user input. The same protocol for the 8 LED row applies, as always.

6. Each recording file will be written to the path /media/... with the naming convention <ID>_<YYYMMDD>T<HHmmssuuuuuu>Z.aiff. All of which can be accessed via the SSD M.2 SATA-to-USB adapter on a PC.

[aidanjohnson]

@leewujung, any and all comments are welcome on the drafted recording regime/strategy. 😃