Not making access file

[EmCNabs]

When I run the following command, it does not create an access file _python3 processfile.py --profile step1 /Directory

The script ends with this:

running command: ffprobe -v error -hide_banner -show_format -show_streams -print_format json "/Users/medialab/Desktop/vrecord_Capture/02_JPV-AV_Ready/2012_79_2_343_1a_PM/2012_79_2_343_1a_PM.mkv" > /Users/medialab/Desktop/vrecord_Capture/02_JPV-AV_Ready/2012_79_2_343_1a_PM/2012_79_2_343_1a_PM_qc_metadata/2012_79_2_343_1a_PM_ffprobe_output.txt
Traceback (most recent call last):
  File "/Users/medialab/github/jpc/videoqc_jpc/JPC_AV/process_file.py", line 362, in <module>
    main()
  File "/Users/medialab/github/jpc/videoqc_jpc/JPC_AV/process_file.py", line 322, in main
    parse_ffprobe(ffprobe_output_path)
  File "/Users/medialab/github/jpc/videoqc_jpc/JPC_AV/ffprobe_check.py", line 83, in parse_ffprobe
    settings_dict1, settings_dict2, settings_dict3 = parse_encoder_settings(encoder_settings)
                                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/medialab/github/jpc/videoqc_jpc/JPC_AV/encoder_settings_check.py", line 46, in parse_encoder_settings
    key, value = pair.split("=")
    ^^^^^^^^^^
ValueError: not enough values to unpack (expected 2, got 1)

Though I can run _python3 makeaccess.py file.mkv separately and it will create the mp4 file

[eddycolloton]

can you run "git status"? I want to see which branch and which commit you are on

[eddycolloton]

also, could you: run ffprobe on the file and provide the output.

in JPC_videoQC/config/command_config.yaml , change the ffprobe check to 'no':

ffprobe:      check_ffprobe: 'no'

Then run python3 process_file.py /Directory Don't use the --profile flag for this test or it will overwrite the command_config.yaml

[EmCNabs]

can you run "git status"? I want to see which branch and which commit you are on

/Users/medialab/github/jpc/videoqc_jpc/JPC_AV
≈:≈ git status
On branch bowser
Your branch is up to date with 'origin/bowser'.

[EmCNabs]

also, could you: run ffprobe on the file and provide the output.

in JPC_videoQC/config/command_config.yaml , change the ffprobe check to 'no':

ffprobe:      check_ffprobe: 'no'

Then run python3 process_file.py /Directory Don't use the --profile flag for this test or it will overwrite the command_config.yaml

/Users/medialab/github/jpc/videoqc_jpc/JPC_AV
≈:≈ ffprobe /Users/medialab/Desktop/vrecord_Capture/02_JPV-AV_Ready/2012_79_2_343_1a_PM/2012_79_2_343_1a_PM.mkv
ffprobe version 7.0 Copyright (c) 2007-2024 the FFmpeg developers
  built with Apple clang version 15.0.0 (clang-1500.1.0.2.5)
  configuration: --prefix=/usr/local/Cellar/ffmpeg/7.0 --enable-shared --enable-pthreads --enable-version3 --cc=clang --host-cflags= --host-ldflags='-Wl,-ld_classic' --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libaribb24 --enable-libbluray --enable-libdav1d --enable-libharfbuzz --enable-libjxl --enable-libmp3lame --enable-libopus --enable-librav1e --enable-librist --enable-librubberband --enable-libsnappy --enable-libsrt --enable-libssh --enable-libsvtav1 --enable-libtesseract --enable-libtheora --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopenvino --enable-libspeex --enable-libsoxr --enable-libzmq --enable-libzimg --disable-libjack --disable-indev=jack --enable-videotoolbox --enable-audiotoolbox
  libavutil      59.  8.100 / 59.  8.100
  libavcodec     61.  3.100 / 61.  3.100
  libavformat    61.  1.100 / 61.  1.100
  libavdevice    61.  1.100 / 61.  1.100
  libavfilter    10.  1.100 / 10.  1.100
  libswscale      8.  1.100 /  8.  1.100
  libswresample   5.  1.100 /  5.  1.100
  libpostproc    58.  1.100 / 58.  1.100
[matroska,webm @ 0x7f8ca48045c0] Could not find codec parameters for stream 2 (Attachment: none): unknown codec
Consider increasing the value for the 'analyzeduration' (0) and 'probesize' (5000000) options
[matroska,webm @ 0x7f8ca48045c0] Could not find codec parameters for stream 3 (Attachment: none): unknown codec
Consider increasing the value for the 'analyzeduration' (0) and 'probesize' (5000000) options
[matroska,webm @ 0x7f8ca48045c0] Could not find codec parameters for stream 4 (Attachment: none): unknown codec
Consider increasing the value for the 'analyzeduration' (0) and 'probesize' (5000000) options
Input #0, matroska,webm, from '/Users/medialab/Desktop/vrecord_Capture/02_JPV-AV_Ready/2012_79_2_343_1a_PM/2012_79_2_343_1a_PM.mkv':
  Metadata:
    creation_time   : 2024-03-20T19:15:46.682565Z
    ENCODER         : Lavf60.3.100
    TITLE           : The Soul of Stax
    COLLECTION      : Pearl Bowser Collection
    CATALOG_NUMBER  : 2012.79.2.343.1a
    DESCRIPTION     : An acclaimed documentary written and directed by Philip Priestley. This film explores the history of Stax Records, a record company founded in the 1960s. Featuring music by soul and blues stars, while chronicling the performers' rise, and the label's even
    DATE_DIGITIZED  : 2024-03-20
    ENCODER_SETTINGS: C=Color, S=Analog, VS= NTSC, A=4:3, T=Sony SVO-5800, T=Blackmagic UltraStudio 4K Mini SN123456, ffmpeg vrecord; in-house, O=FFV1mkv, W=10-bit, M=YUV422p10, N=Emily Nabasny
    ENCODED_BY      : Smithsonian Institution NMAAHC
    ORIGINAL_MEDIA_TYPE: VHS
    DATE_TAGGED     : 2024-04-16
    _TECHNICAL_NOTES: Audio: HiFi Audio, HiFi levels captured. Video: Luminance -1.08db ; Black 5.7 IRE
    VIDEO_STREAM_HASH: fe4e9264e36749fe729344c30f347dfa
    AUDIO_STREAM_HASH: 0357dfed163fbb5fc239016afeb21d2a
  Duration: 01:02:30.01, start: 0.000000, bitrate: 89276 kb/s
  Stream #0:0: Video: ffv1 (FFV1 / 0x31564646), yuv422p10le(tv, smpte170m/smpte170m/bt709, bottom coded first (swapped)), 720x486, SAR 9:10 DAR 4:3, 29.97 fps, 29.97 tbr, 1k tbn
      Metadata:
        ENCODER         : FFV1 version 3
        DURATION        : 01:02:30.013000000
  Stream #0:1: Audio: flac, 48000 Hz, stereo, s32 (24 bit)
      Metadata:
        ENCODER         : Lavc60.3.100 flac
        DURATION        : 01:02:30.000000000
  Stream #0:2: Attachment: none
      Metadata:
        filename        : 2012_79_2_343_1a_PM_capture_options.log
        mimetype        : text/x-log
        title           : Capture options selected by user during vrecord process
  Stream #0:3: Attachment: none
      Metadata:
        filename        : 2012_79_2_343_1a_PM_vrecord_input.log
        mimetype        : text/x-log
        title           : Full FFmpeg output from vrecord capture process
  Stream #0:4: Attachment: none
      Metadata:
        filename        : 2012_79_2_343_1a_VideoConditionReport.pdf
        mimetype        : application/pdf
Unsupported codec with id 0 for input stream 2
Unsupported codec with id 0 for input stream 3
Unsupported codec with id 0 for input stream 4

After making the check_ffprobe 'no' , it did make the access file

/Users/medialab/github/jpc/videoqc_jpc/JPC_AV
≈:≈ python3 process_file.py /Users/medialab/Desktop/vrecord_Capture/02_JPV-AV_Ready/2012_79_2_343_1a_PM
config.yaml sourced from /Users/medialab/github/jpc/videoqc_jpc/config/config.yaml
command_config.yaml sourced from /Users/medialab/github/jpc/videoqc_jpc/config/command_config.yaml
Input directory found: /Users/medialab/Desktop/vrecord_Capture/02_JPV-AV_Ready/2012_79_2_343_1a_PM

The file name '2012_79_2_343_1a_PM.mkv' is valid.

Input video file found: /Users/medialab/Desktop/vrecord_Capture/02_JPV-AV_Ready/2012_79_2_343_1a_PM/2012_79_2_343_1a_PM.mkv

|[●▪▪●]| |[●▪▪●]| |[●▪▪●]|

Now processing /Users/medialab/Desktop/vrecord_Capture/02_JPV-AV_Ready/2012_79_2_343_1a_PM/2012_79_2_343_1a_PM.mkv

Directory name "2012_79_2_343_1a_PM" correctly starts with "2012_79_2_343_1a".

Metadata files will be written to /Users/medialab/Desktop/vrecord_Capture/02_JPV-AV_Ready/2012_79_2_343_1a_PM/2012_79_2_343_1a_PM_qc_metadata

No vrecord files found.

Existing stream hashes found!
Not writing stream hashes to MKV

Generating md5 checksum for 2012_79_2_343_1a_PM.mkv via fixity_check.py:
Calculated md5 checksum is f7efde7c62e7649a03357766c8fdd192

MD5 checksum written to /Users/medialab/Desktop/vrecord_Capture/02_JPV-AV_Ready/2012_79_2_343_1a_PM/2012_79_2_343_1a_PM_2024_04_16_fixity.txt

Using MediaConch policy JPC_AV_NTSC_MKV_2023-11-21.xml
running command: mediaconch -p /Users/medialab/github/jpc/videoqc_jpc/config/JPC_AV_NTSC_MKV_2023-11-21.xml "/Users/medialab/Desktop/vrecord_Capture/02_JPV-AV_Ready/2012_79_2_343_1a_PM/2012_79_2_343_1a_PM.mkv" -oc /Users/medialab/Desktop/vrecord_Capture/02_JPV-AV_Ready/2012_79_2_343_1a_PM/2012_79_2_343_1a_PM_qc_metadata/2012_79_2_343_1a_PM_mediaconch_output.csv

running command: exiftool "/Users/medialab/Desktop/vrecord_Capture/02_JPV-AV_Ready/2012_79_2_343_1a_PM/2012_79_2_343_1a_PM.mkv" > /Users/medialab/Desktop/vrecord_Capture/02_JPV-AV_Ready/2012_79_2_343_1a_PM/2012_79_2_343_1a_PM_qc_metadata/2012_79_2_343_1a_PM_exiftool_output.txt
All specified fields and values found in the Exiftool output.

running command: mediainfo -f "/Users/medialab/Desktop/vrecord_Capture/02_JPV-AV_Ready/2012_79_2_343_1a_PM/2012_79_2_343_1a_PM.mkv" > /Users/medialab/Desktop/vrecord_Capture/02_JPV-AV_Ready/2012_79_2_343_1a_PM/2012_79_2_343_1a_PM_qc_metadata/2012_79_2_343_1a_PM_mediainfo_output.txt
All specified fields and values found in the MediaInfo output.
The specified MediaInfo fields or values for embedded metadata are below:
metadata field in General: ORIGINAL MEDIA TYPE does not exist

running command: ffprobe -v error -hide_banner -show_format -show_streams -print_format json "/Users/medialab/Desktop/vrecord_Capture/02_JPV-AV_Ready/2012_79_2_343_1a_PM/2012_79_2_343_1a_PM.mkv" > /Users/medialab/Desktop/vrecord_Capture/02_JPV-AV_Ready/2012_79_2_343_1a_PM/2012_79_2_343_1a_PM_qc_metadata/2012_79_2_343_1a_PM_ffprobe_output.txt
Running ffmpeg on /Users/medialab/Desktop/vrecord_Capture/02_JPV-AV_Ready/2012_79_2_343_1a_PM/2012_79_2_343_1a_PM.mkv to create access copy /Users/medialab/Desktop/vrecord_Capture/02_JPV-AV_Ready/2012_79_2_343_1a_PM/2012_79_2_343_1a_PM_access.mp4
FFmpeg Access Copy Progress: 100.00%
Please note that any warnings on metadata are just used to help any issues with your file. If they are not relevant at this point in your workflow, just ignore this. Thanks!

Processing complete:
 ___   __   _  ___       ____  ___       ___       ____ _ _   ____      _             ___  __  __ 
|_  ) /  \ / ||_  )     |__  |/ _ \     |_  )     |__ /| | | |__ /     / | __ _      | _ \|  \/  |
 / / | () || | / /        / / \_, /      / /       |_ \|_  _| |_ \     | |/ _` |     |  _/| |\/| |
/___| \__/ |_|/___| ___  /_/   /_/  ___ /___| ___ |___/  |_| |___/ ___ |_|\__,_| ___ |_|  |_|  |_|
                   |___|           |___|     |___|                |___|         |___|             

Output files saved in the directory: /Users/medialab/Desktop/vrecord_Capture/02_JPV-AV_Ready/2012_79_2_343_1a_PM/2012_79_2_343_1a_PM_qc_metadata

Process time for 2012_79_2_343_1a_PM: 
time start: 2024-04-16 20:04:39; time end: 2024-04-16 20:22:46; 
total time: 00:18:06

|[●▪▪●]| |[●▪▪●]| |[●▪▪●]|

::::    ::: ::::    ::::      :::         :::     :::    :::  ::::::::  
:+:+:   :+: +:+:+: :+:+:+   :+: :+:     :+: :+:   :+:    :+: :+:    :+: 
:+:+:+  +:+ +:+ +:+:+ +:+  +:+   +:+   +:+   +:+  +:+    +:+ +:+        
+#+ +:+ +#+ +#+  +:+  +#+ +#++:++#++: +#++:++#++: +#++:++#++ +#+        
+#+  +#+#+# +#+       +#+ +#+     +#+ +#+     +#+ +#+    +#+ +#+        
#+#   #+#+# #+#       #+# #+#     #+# #+#     #+# #+#    #+# #+#    #+# 
###    #### ###       ### ###     ### ###     ### ###    ###  ########  

All files processed!

Overall processing time for all directories: 00:18:07

[eddycolloton]

ok, looks like the ffprobe check is too rigid for Bowser data. I'll look into it. For now, maybe just keep check_ffprobe on 'no'. I can alter the profiles to keep the ffprobe output check off. Do you use step1 and step2 profile?

[EmCNabs]

so far we have just been using the step1 profile

[eddycolloton]

Got it. As of last commit, both step1 and step2 on the Bowser branch do not perform a ffprobe check. step1 will still make the ffprobe sidecar.

As we develop the scripts, it will be good to know which combination of checks are helpful for this use case. If you ever want to change either of the profiles, you can edit yaml_profiles.py. If there's a combination of checks you use a lot, we can make a new profile for that.

The profiles part of yaml_profiles is pretty straightforward, simply change the checks you want saved to a particular profile:

profile_step1 = {
    "tools": {
        "qctools": {
            "run_qctools": 'no',
            "check_qctools": 'no'   
        },

Just be careful to mind quotation marks and commas when editing, python is very picky about that sort of thing

[eddycolloton]

I need to fix how the script parses the "encoder settings." This stuff: C=Color, S=Analog, VS= NTSC, A=4:3, T=Sony SVO-5800, T=Blackmagic UltraStudio 4K Mini SN123456, ffmpeg vrecord; in-house, O=FFV1mkv, W=10-bit, M=YUV422p10, N=Emily Nabasny

Where/When do the encoder settings get entered?

Where are the meaning of the letters defined? For example what does "T=" mean, and does it mean the same thing when it equals the VHS deck as it does when it equals the blackmagic ultrastudio?

[BleakleyMcD]

@eddycolloton the encoder settings are usually entered at the same time all the other metadata is embedded. When exactly that is is depends on what specific workflow is happening.

The meaning of the letters is from FADGI and @EmCNabs did a great job of modifying the ones available for BWF and DPX into something for video. But, I agree, the FADGI method it is not great on the eyes and hard to understand without referencing FADGI documentation. From FADGI

A=PCM,F=96000,W=24,M=mono,T=Pyramix1; SN16986,
A=PCM,F=96000,W=24,M=mono,T=Lynx; AES16; DIO, (* see note below
about EOL comma use)
Explanation:
Line 1 reads: an analog, mono, open-reel tape played back on a Studer 816 tape
machine with serial number 1007 at tape speed 15 ips.
Line 2 reads: tape was digitized to PCM coding in mono mode at 96 kHz
sampling frequency and 24 bits per sample on a Pyramix 1 DAW with serial
number 16986.
Line 3 reads: the audio was stored as a BWF file with PCM coding in mono mode
at 96 kHz sampling frequency and 24 bits per sample using a Lynx AES16 digital
input/output interface
NOTE: These examples from the Sound Directions project include a comma (“,”)
at the end of each line of text but the EOL comma is not included in EBU R98.
FADGI is including the comma in this document to faithfully represent the Sound
Directions example but FADGI does not require EOL commas.

mkv is much more flexible so I'm very open to modifying this to something... easier.

[eddycolloton]

Ah, knowing that the line breaks having meaning does help me understand. Without those, it is hard to programmatically parse which letters pertain to which "group."

Can we add some indication that we're switching to describing something else? Is it is always in the same order? Source, transfer, destination file?

I was going to propose a semi-colon but looks like we are already using that in "T=" for the transfer: T=Blackmagic UltraStudio 4K Mini SN123456, ffmpeg vrecord; in-house

Could we wrap groups in brackets? Not sure I'm grouping correctly: [C=Color, S=Analog, VS= NTSC, A=4:3, T=Sony SVO-5800] [T=Blackmagic UltraStudio 4K Mini SN123456, ffmpeg vrecord; in-house] [O=FFV1mkv, W=10-bit, M=YUV422p10, N=Emily Nabasny]

[BleakleyMcD]

@eddycolloton can do any of the above, yes. will have a new mkvnote today. let's chat next week. I'll open a new issue for this.