search

Telecommunication-Telemedia-Assessment / quat

quat -- quality analysis tools
https://telecommunication-telemedia-assessment.github.io/quat/index.html
GNU General Public License v3.0
0 stars 1 forks source link

temporal pooling for brisque features #5

Closed Ali-Flt closed 2 years ago

Ali-Flt commented 2 years ago

By reading your advanced_pooling function in quat/video.py I couldn't understand how you're handling features such as brisque that are a list of several features. Also running the code for the brisque feature caused the below error so I couldn't guess the behavior from the run result either. 

Traceback (most recent call last):

  File "~/anaconda3/envs/devEnv/lib/python3.10/site-packages/spyder_kernels/py3compat.py", line 356, in compat_exec
    exec(code, globals, locals)

  File "~/workspace/features_extract/feature_extract.py", line 57, in <module>
    poooled, full_report = extract_features_full_ref(

  File "~/workspace/features_extract/pixelmodels_2/pixelmodels/common.py", line 377, in extract_features_full_ref
    pooled_features, full_features = __store_and_pool_features(dis_video, features, meta, features_temp_folder)

  File "~/workspace/features_extract/pixelmodels_2/pixelmodels/common.py", line 261, in __store_and_pool_features
    pooled_features = dict(advanced_pooling(values, name=f), **pooled_features)

  File "~/anaconda3/envs/devEnv/lib/python3.10/site-packages/quat/video.py", line 128, in advanced_pooling
    **advanced_pooling(

  File "~/anaconda3/envs/devEnv/lib/python3.10/site-packages/quat/video.py", line 136, in advanced_pooling
    values = values[~np.isnan(values)]

TypeError: ufunc 'isnan' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''

So I'm just curious to know whether you tend to treat each of the 36 features as an independent one or if you perform a mean over all 36 features to create a single one (or any other approach)?

stg7 commented 2 years ago

well, in theory, the advanced pooling method is called for each part individually, so when brisque returns a list per frame, it creates for each per frame feature with index $I$ a pooled view as brisque_$I$, this method seems to work fine, the problem here is somehow related to the full_ref part (because nofu uses also brisque as a feature), I will check, where the problem is.

however, important to mention, is that when the features are adjusted, the model needs to be re-trained, because it will otherwise end up in a "misshape" of the feature values (because the model has been trained with a different feature set)

stg7 commented 2 years ago

I added now a specific "unnesting" function to also allow brisque as a feature (due to the nesting of values pandas cant create there a dataframe), this should solve the issue, tests are ongoing.

stg7 commented 2 years ago

I did a check to include brisque in hyfr, and the feature extraction now works fine