mkrusche commented 5 years ago

Review Bitcurator to determine what it does.

https://bitcurator.net/bitcurator/ https://github.com/BitCurator/bitcurator-access/wiki https://confluence.educopia.org/display/BC https://github.com/bitcurator

https://bitcurator.github.io/

nkmeyers commented 5 years ago

Bitcurator contact: Cal Lee at UNC https://sils.unc.edu/people/faculty/profiles/Cal-Lee Bitcurator contact Kam Woods at [UNC https://github.com/kamwoods ](UNC https://github.com/kamwoods )

dbrower commented 5 years ago

BitCurator Evaluation

Initial review of BC from website. From BC wiki page:

The BitCurator Environment is a Ubuntu-derived Linux distribution geared towards the needs of archivists and librarians. It includes a suite of open source digital forensics and data analysis tools to help collecting institutions process born-digital materials. BitCurator supports positive digital preservation outcomes using software (see our Tasks and Tools page) and practices adopted from the digital forensics community.

In the BitCurator Environment you can:

Create forensic disk images: Disk images packaged with metadata about devices, file systems, and the creation process.

Analyze files and file systems: View details on file system contents from a wide variety of file systems.

Extract file system metadata: File system metadata is a critical link in the chain of custody and in records of provenance.

Identify sensitive information: Locate private and sensitive information on digital media and prepare materials for access.

Locate and remove duplicate files: Know what files to keep and what can be discarded.

BC runs in its own virtual machine. It packages many other tools together. It also has some custom-developed tools.

Custom developed tools

BitCurator Reporting Tool A GUI-driven (and optionally command-line) tool for running forensics tools in sequence to produce human- and machine-readable reports.
BitCurator Disk Image Access Tool A GUI interface to browse raw and forensically-packaged disk images, export files and deleted items, and view disk image metadata.
BitCurator Mounter A Qt GUI application to list currently attached devices along with technical details. Allows users to mount fixed and removable media according to the current mount policy.
BitCurator Read-Only AppIndicator A Ubuntu AppIndicator allowing users to switch the system mount policy between "Read Only" and "Read/Write" for any attached media prior to mounting.
Nautilus scripts Support for various interactions with files and file systems.
Safe Mount Software write-blocking for digital media.

Packaged external tools

These are tools that are not developed by the BitCurator Consortium itself, but are included in the BC distribution.

Disk imaging

Guymager: Multi-threaded open-source forensic disk imaging tool.
dcfldd: A forensics-focused rewrite of dd.
dd: Create raw disk images and transfer data between devices.
ddrescue: A version of dd with additional options for data recovery.
ewfacquire: Acquire Expert Witness packaged disk images from devices on the command line.
cdrdao: CD imaging tool (primarily for audio CDs).

Forensic analysis

bulk_extractor: A stream-based tool for disk image analysis.
bulk_extractor Viewer (BEViewer): The GUI front-end for bulk-extractor
DFXML tools: A set of C and Python programs to process Digital Forensics XML.
fiwalk: File system analysis and DFXML export.
The Sleuth Kit: A suite of forensics tools, utilities, and APIs.
libewf: Open-source support for the Expert Witness format.
AFFLIB: Open-source library for the Advanced Forensic Format.
pyExifToolGUI: A GUI front-end for Exiftool. Allows editing of image metadata.
sdhash: File similarity tool using similarity digests.
ssdeep: Fast hash generation.

Other tools

ClamAV / ClamTK: Virus scanning.
FSlint: Duplicate file identification and deletion.
HFS Utilities: Utilities providing access to legacy HFS file systems, such as HFS Explorer.
readpst: A utility for reading and exporting the contents of PST files.
GTK Hash: A cryptographic hashing tool.
GHex: A hex viewer/editor

Thoughts

It appears BC is intended to be used by humans. It is unclear how many of the custom developed interfaces can be interacted with programmatically, especially if the distribution is made to run in a VM (not sure whether this is for security or because most of these tools are linux based, and BC wants to work on Windows). However, many of its tools may be reusable, and may have command-line forms that would facilitate automation. (comment from BC wiki: "The virtual machine version of BitCurator is useful for testing and experimentation, but it is recommended that you run BitCurator on a dedicated machine in production environments by installing from the Live ISO image.")

It does appear like they have done a good job of curating such tools, so if any kind of file analysis functionality is needed, it would be useful to look here for any suggestions.

That said, some standard digital library tools for file identification are not listed, e.g. DROID, file, FITS, PRONOM.

mkrusche commented 5 years ago

Don presented his findings to the team in the sprint review on 4/28. The PI's will get together to discuss them.