Script that a user could use at the command line to conduct searches

[technosaby]

Suppose a user wanted to find examples of a drum roll, or laughter, or a helicopter, or a drum roll and laughter within the same second, or a drum roll and a helicopter within the same 2 seconds. What would be the input command to the bash script, and what would be the output? The form of the input needs to be straightforward. Let’s call this bashscript ssfx for “search for sound effects.” Suppose it has flags --files for files to search; it would need to be able to accept all files in a directory, and recursively —effects effects —within interval in seconds within which range is satisfied

A user would command something like ssfx -r —files /tv/2022/2022-03/20220305 —effects{{drum|percussion}&{laughter}} —within 5

Then ssfx would egrep through the directory 20220305 recursively looking for cases where, within a 5 second interval, there is metadata for laughter AND (Either drum or percussion), and it would output its hits, including name of the file found effects time interval in which they are found. some way to play that interval (with perhaps variable buffer on either side) in some kind of player. The output should be in a csv file for easy handling, and for import to R, so the user could run statistics.

[technosaby]

@brucearctor @turnermarkb I want to understand "—effects{{drum|percussion}&{laughter}} —within 5" of the comment. Can you give me an example from this file.

For example in this file, if I give a condition {Speech|Conversation}&{Clicking}, what will be the output ? 2010-01-01_2335_US_CSPAN2_World_War_II copy.txt

<<Here is the section which contains the tags, Full file also attached>> "201001012335024.000|201001012335024.975|SFX_01|{'Speech': 1.0, 'Narration, monologue': 0.01, 'Inside, small room': 0.01} 201001012335024.400|201001012335025.375|SFX_01|{'Speech': 1.0, 'Conversation': 0.02, 'Narration, monologue': 0.01, 'Inside, small room': 0.01} 201001012335024.800|201001012335025.775|SFX_01|{'Speech': 0.16, 'Narration, monologue': 0.01, 'Clicking': 0.03, 'Silence': 0.41, 'Inside, small room': 0.01} 201001012335025.200|201001012335026.175|SFX_01|{'Speech': 0.95, 'Clicking': 0.01, 'Inside, small room': 0.02} 201001012335025.600|201001012335026.575|SFX_01|{'Speech': 1.0} 201001012335026.000|201001012335026.975|SFX_01|{'Speech': 0.99} 201001012335026.400|201001012335027.375|SFX_01|{'Speech': 1.0} 201001012335026.800|201001012335027.775|SFX_01|{'Speech': 0.99} 201001012335027.200|201001012335028.175|SFX_01|{'Speech': 1.0, 'Narration, monologue': 0.01} 201001012335027.600|201001012335028.575|SFX_01|{'Silence': 1.0} 201001012335028.000|201001012335028.975|SFX_01|{'Speech': 0.91, 'Narration, monologue': 0.01, 'Clicking': 0.02, 'Silence': 0.03, 'Inside, small room': 0.02} 201001012335028.400|201001012335029.375|SFX_01|{'Speech': 1.0} 201001012335028.800|201001012335029.775|SFX_01|{'Speech': 1.0}"

As the file contains time for almost 1 sec each line, shall I have to do the search like this, 0-5th second, 1-6th second, 2-7th second, 3-8th second .... and then filter the results of the tags. But this result will also contain common blocks, for example if the filter matched at 4th second, the result will come for 0-5, 1-6, 2-7, 3-8, 4-9th second. Then I have to filter the results for a unique slot?

[brucearctor]

I understood the idea to output something like the initial timestamp.

There would presumably be an additional parameter, ex: threshold of condition... but to the question at hand, that is [ currently ] not a major consideration.

The initial timestamp(s) for each series is useful because it allows someone to then view the result at those time(s).

[technosaby]

@brucearctor I updated my comment above to give more clarity on the question. Can you please check?

[brucearctor]

I think Mark's point was not to specify or even that developing the ideal searching algorithm was critical, but to ensure that you're thinking about how the data can/would-likely get used.

Also, at the moment, the ability to search at all is hugely valuable -- there is then room to worry about making it more efficient [ better searching algorithm ].

[technosaby]

@brucearctor @turnermarkb Can you please take a look at this for the changes? It will be great if you can go through the documentation and sample files and let me know if they are okay.

[technosaby]

@brucearctor As suggested by you, I am using jq for parsing the filtering logic. For now, I am focussing on filtering in one time frame like, For this frame output scores, [{"Music": "0.89", "Background music": "0.01", "Theme music": "0.02", "Sound effect": "0.01", "Television": "0.13"}] if I add filter as "Music & Television", then it will give, "Music": "0.89", "Television": "0.13". Also can I take the input in the same format as jq filters or I need to take them in normal logic gates and convert to jq ?

[brucearctor]

I think the point is to have some example ways to query. jq not a requirement, but a pretty convenient one, if has the structure like: {"Music": "0.89", "Background music": "0.01", "Theme music": "0.02", "Sound effect": "0.01", "Television": "0.13"} and using bash/zsh/etc.

As to the eventual output "Music": "0.89", "Television": "0.13". seems OK, that can always be piped into something else that combines, in the event that is desired.

Ultimately, as I understand it, point is that most users will have no idea how to search the data, so the intent was to at least provide some things that people might be able to use [ fill in variables ], and leave the door open to extending in the future.

[technosaby]

@brucearctor I think jq is very useful for our case and I also integrated with my ssfx to get some output. Just a technical question.

For a sample JSON string {"Music": "0.89", "Background music": "0.01", "Theme music": "0.02", "Sound effect": "0.01", "Television": "0.13"} when I give this command jq '.[] | .Music , .Television' it returns 0.89, 0.13. Is there any way to return the keys along with the values like {"Music": "0.89", "Television": "0.13"}

[brucearctor]

I don't have the commands off the top of my head. But, I imagine there is a way to do what you're thinking, it may start to get somewhat complex [ but once figured out, then quite usable ... ] :-)

[technosaby]

@brucearctor @turnermarkb I have a first draft for the changes which we discussed last week

• Adding the TOP section for files without seg
• Modifying Codebook
• A script to filter tags based on queries

It will be great if you can review the code and let me know your comments in the PR