Create file identifiers

[tarakc02]

using sha1 hash. since there's so much data, this will take a few minutes to run. it would be nice to not have to rehash unchanged files if/when there are new files and we have to update the task.

[ayyubibrahimi]

How should we begin this step? I anticipate being set-up/having had figured out how to work remotely by tomorrow now that I'm up on Eleanor with the ability to pull data.

[tarakc02]

we can focus for now on the basic requirement, and ignore the note about "it would be nice ...". So basically we want to output a dataframe with three columns:

• the full path to the file in question starting from the root of the repo, so something like index-files/input/wrongful-convictions-docs/Abc_Def_XXXX/.../filename.ext
• the sha1 hash of the file's contents (when using python's hashlib, the algo is "sha1"). you can test your code on a file, and confirm that you get the same hash value as if you're on the terminal and do sha1sum path/to/filename.ext
• the unique identifier obtained by taking the first 8 characters of the sha1 hash calculated in the previous step

Since version 3.11, python's hashlib includes a convenient file_digest method. On eleanor, you should be able to activate the ipno-exonerations conda environment, which has python3.11 set up. Alternatively, this stack overflow page looks like it has examples.