Profiles & Formats (Notion & Airtable)

[IAmVigneswaran]

Continuing the discussion here from here.

Current default format is .csv (For Notion Database)

This could add further complexities. What if there were different "Profiles" for output formats for different database platforms, NLEs and DAWs. For each profiles, the output data extension and structure would formatted in a different way.

Example: For Notion - the current .csv format and structure. For ProTools - it could be in a text file that ProTools could import. For Resolve - it could be in a EDL file.

[orchetect]

This is definitely possible.

The refactors I've completed so far have separated generic logic form format-specific logic.

All of the file reading/parsing and thumbnail output routines are generic (but could have custom iterators or implementations if needed). Routines that specifically format data for the csv2notion "profile" are clearly abstracted away.

[IAmVigneswaran]

Having other formats will really open up the interaction with with other application. Users of FCP could send Marker Data information to users other application (ProTools, Resolve, Etc).

Example : --proflie-format <Notion, Airtable, ProTools, Resolve>

For some profiles like ProTools, we can turn off image extraction?

We can add more profiles of most major DAWs and NLEs based on user's input.

[IAmVigneswaran]

@orchetect Can simplicity we rename the current profile has Notion instead of csv2notion?

For 0.2 release, we could also add Airtable profile has well?

For Airtable profile, it would be the identical as Notion. The only difference, it won't generate Marker Icons, and it won't include 'Icon Image' column for the .csv file.

[orchetect]

we could also add Airtable profile has well?

Sure, I think notion and airtable would be a good place to start for profiles.

CSV is just an implementation detail like we've talked about, so it would be good to refer less to "CSV" directly in the CLI help text and refer to a more abstract term like "metadata file" or "manifest file". Because different profiles may use different file formats besides CSV.

[orchetect]

On the whole, the more profiles we add (#21, #23, #24, #25, #26, #27), the more MarkersExtractor becomes a general-purpose conversion utility. I'm not sure how far down that road you want to go.

The ones that can take advantage of the generic processing routines that have already been written are certainly easier to implement and could be implemented sooner.

As for specific targets and file types, they would need to be researched and built (and possibly reverse-engineered).

Along those lines, DAWFileKit (as its name might suggest) implements a few import/export file formats from DAWs and new formats as they are developed could be added to it. Similar to how we're using pipeline as a dependency for FCPXML (since it's such a heavy format it generally deserves a library all its own), DAWFileKit can process various other file types and distils parsed data into format-agnostic data prototypes that can be converted to other encodings. (I've considered renaming it NLEFileKit or TimelineFileKit or something since it's taking on more video editing formats as of recent and isn't necessarily just DAW-centric.)

[IAmVigneswaran]

On the whole, the more profiles we add (https://github.com/TheAcharya/MarkersExtractor/issues/21, https://github.com/TheAcharya/MarkersExtractor/issues/23, https://github.com/TheAcharya/MarkersExtractor/issues/24, https://github.com/TheAcharya/MarkersExtractor/issues/25, https://github.com/TheAcharya/MarkersExtractor/issues/26, https://github.com/TheAcharya/MarkersExtractor/issues/27), the more MarkersExtractor becomes a general-purpose conversion utility. I'm not sure how far down that road you want to go.

True! The current absolute purpose of this tool to send to databases like Notion or Airtable. Or even Filemaker.

My only intention of opening these additional issues was based on our discussions with your suggestion to future proof the output format beyond the constrains of .csv or confines of csv2notion tool.

Yes, I do understand on how far we could support. Or if there is even a user workflow needed for such a feature. It is all based on user feedback and request.

The ones that can take advantage of the generic processing routines that have already been written are certainly easier to implement and could be implemented sooner.

We could attempt on those general purpose and text formats?

Along those lines, DAWFileKit (as its name might suggest) implements a few import/export file formats from DAWs and new formats as they are developed could be added to it. Similar to how we're using pipeline as a dependency for FCPXML (since it's such a heavy format it generally deserves a library all its own), DAWFileKit can process various other file types and distils parsed data into format-agnostic data prototypes that can be converted to other encodings. (I've considered renaming it NLEFileKit or TimelineFileKit or something since it's taking on more video editing formats as of recent and isn't necessarily just DAW-centric.)

Interesting! Thanks for sharing.

[orchetect]

My only intention of opening these additional issues was based on our discussions with your suggestion to future proof the output format beyond the constrains of .csv or confines of csv2notion tool.

Absolutely. So even if we only implement one or two workflows at the current time, it's valuable for the codebase to be structured in a way where additional formats could be easier to add in future (by myself or someone else).

[orchetect]

The four .png marker icon assets that we have - are those specific to Notion or is the intention to reuse them for other profiles?

[IAmVigneswaran]

The four .png marker icon assets that we have - are those specific to Notion or is the intention to reuse them for other profiles?

Yes, they are specifically for Notion and csv2Notion. csv2Notion has the ability to batch upload icons for Notion.

[orchetect]

• [x] Rename 'csv2notion' profile to 'notion'
• [ ] Further refactor codebase to modularize generic methods used by profiles
• [x] Add 'airtable' profile

[orchetect]

Are we also still exporting thumbnails for Airtable?

[IAmVigneswaran]

Are we also still exporting thumbnails for Airtable?

Yes. Airtable supports full gallery view.

https://support.airtable.com/docs/guide-to-gallery-view https://support.airtable.com/docs/viewing-attachments-in-airtable

[orchetect]

Really cool. Just out of curiosity, is there a native way to import the CSV/thumbnails to Airtable or are you using a special script/utility?

[IAmVigneswaran]

Really cool. Just out of curiosity, is there a native way to import the CSV/thumbnails to Airtable or are you using a special script/utility?

AFAIK, there is no sleek user friendly tool like csv2notion for Airtable.

However, there is the official CSV import extension within Airtable. And yes, you would need to write a small piece of script to batch link the images to the individual records.

[orchetect]

Currently standard markers and chapter markers output "Not Started" for the "Status" field. Is this a special consideration for Notion?

I only ask because standard/chapter markers are not To Do markers and "Not Started" is a bit confusing because it's not applicable. Would it be a problem if this field is left blank for standard/chapter markers?

[IAmVigneswaran]

Currently standard markers and chapter markers output "Not Started" for the "Status" field. Is this a special consideration for Notion?

Yes. The status column is for Notion Status. csv2notion has the ability to detect a column that is tagged for Notion Status property.

I only ask because standard/chapter markers are not To Do markers and "Not Started" is a bit confusing because it's not applicable. Would it be a problem if this field is left blank for standard/chapter markers?

The values can't be set blank in order for "status column" to work. Standard and Chapter Markers should be set to "Not Started". Users can obviously customise the values of the Notion Status property within Notion. In order for the upload process to work for all users, we need to match it with the default values.

[orchetect]

Took a fair bit more internal refactors but multiple profile support is now implemented with Notion and Airtable profiles.

Will be in next alpha build.

[IAmVigneswaran]

@orchetect Right now the *.json output is in this following manner.

{
    "Event Name" : "Demo 1",
    "Clip Name" : "Nature Makes You Happy",
    "Project Name" : "Marker Data Demo_V2",
    "Library Name" : "Markers",
    "Video Role & Subrole" : "Not Assigned",
    "Marker ID" : "Marker Data Demo_V2_00:00:20:16",
    "Image Filename" : "Marker Data Demo_V2_00_00_20_16.png",
    "Clip Duration" : "00:02:22:01",
    "Marker Position" : "00:00:20:16",
    "Audio Role & Subrole" : "Dialogue",
    "Status" : "In Progress",
    "Type" : "To Do",
    "Notes" : "Note Test 1",
    "Marker Name" : "(To-Do) Penguin",
    "Clip Filename" : "Nature Makes You Happy.mp4",
    "Checked" : "false"
  },

Can we list the entires in the order of the .csv?

{
    "Marker ID" : "Marker Data Demo_V2_00:00:20:16",
    "Marker Name" : "(To-Do) Penguin",
    "Type" : "To Do",
    "Checked" : "false"
    "Status" : "In Progress",
    "Notes" : "Note Test 1",
    "Marker Position" : "00:00:20:16",
    "Clip Name" : "Nature Makes You Happy",
    "Clip Filename" : "Nature Makes You Happy.mp4",
    "Clip Duration" : "00:02:22:01",
    "Video Role & Subrole" : "Not Assigned",
    "Audio Role & Subrole" : "Dialogue",
    "Event Name" : "Demo 1",
    "Project Name" : "Marker Data Demo_V2",
    "Library Name" : "Markers",
    "Image Filename" : "Marker Data Demo_V2_00_00_20_16.png",
  },

Thank you.

[orchetect]

That's not really how JSON works. It's dictionary based which doesn't sort. If you're parsing it, use a JSON decoder and then read it by key/value.

[IAmVigneswaran]

Thanks for the information. I will pass the information to the developer.

[IAmVigneswaran]

My bad!

Can we enable json output for both Notion and Airtable profiles in the next build.

We can test with uploading to Notion using CSV2Notion Neo.

https://github.com/TheAcharya/csv2notion-neo/issues/9

Thank you.

[orchetect]

Sure, but probably for 0.2.9.

For the Audio Role & Subrole field, instead of its value being a single string, it will be a JSON array of strings. This is because there can be more than one audio role for a marker/caption and will make parsing easier. As mentioned in the other issue thread (https://github.com/TheAcharya/MarkersExtractor/issues/66#issuecomment-1833168552), we can't use commas to separate roles in a single string since FCP allows role names to contain commas.

[IAmVigneswaran]

we can't use commas to separate roles in a single string since FCP allows role names to contain commas.

True!

We will try and update both Airlift and CSV2Notion Neo to support the new separator for parsing.

Are we using <NULL> or \n for the separator?

Since we are going full on with .json and not bounded by the limitation of .csv.

What is the best separator?

Could they be quotes like "Dialogue.MixL","Dialogue.MixR","Dialogue.Boom-1"

[orchetect]

CSV currently uses comma without a space (,) as a delimiter for multiple roles but is not ideal so it could be changed to null \n. Quotes does not help and makes reading the CSV more complex. I think quotes are also allowed in role names in FCP so it gets pretty crazy.

For JSON we won't need delimiters. Audio Role & Subrole will be an ordered array, not a single concatenated string. That's the point of preferring JSON. It's more versatile of a data format. It encodes strings so commas and quotes can be used within a string, and they will decode properly without any extra effort.

{
    "Marker ID" : "Marker Data Demo_V2_00:00:20:16",
    ... etc ...
    "Video Role & Subrole" : "Not Assigned",
    "Audio Role & Subrole" : [
        "Dialogue.MixL",
        "Dialogue.MixR",
        "Role, With Comma.Role, With Comma-1",
        "\"Quoted\".\"Quoted\"-1"
    ],
    ... etc ...
},

Based on these FCP audio roles:

Dialogue.MixL
Dialogue.MixR
Role, With Comma.Role, With Comma-1
"Quoted"."Quoted"-1

[IAmVigneswaran]

CSV currently uses comma without a space (,) as a delimiter for multiple roles but is not ideal so it could be changed to null \n. Quotes does not help and makes reading the CSV more complex.

Noted!

Since we have decided JSON for both Notion and Airtable. Does it make sense to have a 4th Export Profile just for .csv.?

We can just called it CSV. Just for compatibility with spreadsheet applications.

[orchetect]

We could do that.

Some fields have been formatted specifically with Notion in mind. I feel like it's only coincidental that Airtable is using the same fields and formats.

I would wonder if a generic CSV profile was offered that we would remove some of the choices/assumptions about formatting that we made for Notion. Or we just leave the data as it is now.

Either way, it's best to create a new issue for a CSV profile if we want that.

[IAmVigneswaran]

Noted! Hope we can get JSON working for CSV2Notion Neo within next 2 weeks.

Once that is sorted, we can formalise a generic CSV profile.

[IAmVigneswaran]

We have implemented JSON support to CSV2Notion Neo based on previous build of MarkersExtractor (0.2.3) using Airtable Profile.

https://github.com/TheAcharya/csv2notion-neo/actions/runs/7066755647

Have not tested with nested values / ordered arrays yet.

[orchetect]

@milanvarady Notion and Airtable profiles now both output CSV and JSON.

The field data for JSON is identical to the CSV except for the audioRole.

• In CSV, audioRole is still a single string that is a comma-separated list of audio roles.
• In JSON, audioRole is an ordered array of strings as described above.

Will be in 0.2.9.

[IAmVigneswaran]

But in 0.3.0, let's remove CSV output for both Notion and Airtable profiles? This will prevent any confusion?

Considering #74, we can have separate single CSV and/plus TSV profile? For compatibility with excel and google sheets.

[orchetect]

Once you've got your tooling moved over to parsing the JSON manifests then we can transition to dedicated CSV/TSV profiles.

[IAmVigneswaran]

Now Airlift fully supports parsing of MarkersExtractor's .json data. https://github.com/TheAcharya/Airlift/releases/tag/v1.0.4

All the nested roles in are now interpreted as Multi-Select column type in Airtable.

Next on the table is updating CSV2Notion Neo.

[IAmVigneswaran]

Now CSV2Notion Neo fully supports parsing of MarkersExtractor's .json data. https://github.com/TheAcharya/csv2notion-neo/releases/tag/v1.1.0

All the nested roles in are now interpreted as Multi-Select column type in Notion Database.

Any tweaks in future easily be addressed in both Airlift and CSV2Notion Neo tools.

We can now drop .csv file in both Notion and Airtable Profile and create #74.

[orchetect]

Removed CSV manifest file export for Airtable and Notion profiles. Change will be in 0.3.1.