search

faroit / python_audio_loading_benchmark

Benchmark popular audio i/o packages
138 stars 10 forks source link

tensorflow2, librosa8, better plots #8

Closed faroit closed 3 years ago

faroit commented 4 years ago

Bug fixes

bmcfee commented 4 years ago

Gentle nudge on this one, since the reported librosa benchmarks are now two majors out of date :grin:

faroit commented 4 years ago

Gentle nudge on this one, since the reported librosa benchmarks are now two majors out of date 😁

sorry again especially, since I am aware that this should be addressed soon as I never wanted to make librosa look bad in this... That being said, I have two main issues that I could need could get some help with, as I am still lacking time to finish this up... Basically you mentioned two things here:

  1. Use timit to launch individual file loadings to reduce warm-up bias: I will propose something, maybe you could review that.

  2. Change the plotting to swarm plots: I played with that but I wasn't that successful in getting something that brings a clear message to viewers. I could provide the pandas files so you may play with that

  3. tf.data pipelines aren't really playing nice with errors (such as when a file format is not supported) using

dataset = dataset.apply(tf.data.experimental.ignore_errors()) results in no errors at all, thus making it even even harder to track down if the data was loaded successfully. So I guess I have to check the actual return value... any other idea?

faroit commented 4 years ago

@bmcfee regarding 3. I guess the easiest is to have fixed matrix of supported pairs of codecs and libs that will be tested instead of using try/except...

bmcfee commented 4 years ago 
1. I will propose something, maybe you could review that.

Sure, happy to!

2\. I played with that but I wasn't that successful in getting something that brings a clear message to viewers. I could provide the pandas files so you may play with that

That's fine -- it was just a suggestion. I generally have a preference toward raw data over aggregates, but if it's too unwieldy here, there's no problem leaving it as is.

results in no errors at all, thus making it even even harder to track down if the data was loaded successfully. So I guess I have to check the actual return value... any other idea?

Hrm, indeed. I guess you could do a length check (expected # of samples) if you don't want to do a full inspection of the data.

I guess the easiest is to have fixed matrix of supported pairs of codecs and libs that will be tested instead of using try/except...

That's probably a safer option, but a lot more work I'm guessing.

faroit commented 4 years ago

@bmcfee a problem with 1. is that tf.data pipelines need to be compiled for each file. That would make the comparison between frameworks (np vs torch vs tf) useless as tf would probably be last. Still it would be meaningful to give relative results: aka. which loading method is the fasted for each tensor lib. Would you say this is worth it then?

bmcfee commented 4 years ago

That would make the comparison between frameworks (np vs torch vs tf) useless as tf would probably be last. Still it would be meaningful to give relative results: aka. which loading method is the fasted for each tensor lib. Would you say this is worth it then?

Yeah, that's tricky. Maybe worth reporting two versions in that case, one of which factors out the compilation time?

faroit commented 3 years ago

@bmcfee i think its better to merge this now than waiting another year... sorry for this. I didn't had time to polish the plots but all libs are up to date and i am using timit to get a better average

bmcfee commented 3 years ago

Thanks @faroit ! I'm just happy to see the version updates. :grin: