Issues create custom data pipeline

[connormeaton]

I am trying to feed my own videos/groundtruth into this pipeline and I cannot figure out what isn't working.

I created a dataset like this:

    Sample dataset structure:
    -----------------
        SAMPLE/
        |    |-- PARTNERDIR
        |        |-- videoSample.mp4
        |        |-- signalGT.csv
        |    ...

Here is my custom dataset file, named 'sample.py':

import csv
import numpy as np
import pandas as pd
from pyVHR.datasets.dataset import Dataset
from pyVHR.signals.bvp import BVPsignal

class SAMPLE(Dataset):
    """
    Sample dataset structure:
    -----------------
        SAMPLE/
        |    |-- PARTNERDIR
        |        |-- videoSample.mp4
        |        |-- signalGT.csv
        |    ...
    """
    name = 'SAMPLE'
    videodataDIR = '../sampledata/'  # path relative to notebook dir in pyVHR filesystem on GitHub
    BVPdataDIR = '../sampledata/'    # path relative to notebook dir in pyVHR filesystem on GitHub
    signalGT = 'BVP'          # GT signal type
    numLevels = 1             # depth of the filesystem collecting video and BVP files
    numSubjects = 1          # number of subjects
    video_EXT = 'mp4'         # extension of the video files
    frameRate = 25            # vieo frame rate
    VIDEO_SUBSTRING = 'cv_camera' # substring contained in the filename
    SIG_EXT = 'csv'           # extension of the BVP files
    SIG_SUBSTRING = ''      # substring contained in the filename
    SIG_SampleRate = 2000       # sample rate of the BVP files

    def readSigfile(self, filename):
        """ Load BVP signal.
            Must return a 1-dim (row array) signal
        """
        gtTime = []
        gtHR = []
        df = pd.read_csv(filename, sep='\t')
        df = df.dropna()

        cols = []
        cols_ = []
        count = 0
        cols
        for i in df.columns:
            if 'bpm' in i.lower():
                cols.append(i)
        if len(cols) > 2:
            for j in cols:
                    if 'ppg' in j.lower():
                        cols_.append(j)
            cols = cols_
        df = df[cols]
        gtHR = df[0]
        gtTime = df["b'time (s)'"]

        time = np.array(gtTime)
        hr = np.array(gtHR)
        self.SIG_SampleRate = np.round(1/np.mean(np.diff(time)))

        '''import matplotlib.pyplot as plt
        plt.plot(hr)
        plt.show()'''

        return BVPsignal(data, self.SIG_SampleRate)

Here is the config file for it, named sample.cfg:

# default_test.cfg - default test configuration file for TestSuite class

## Default parameters
#
#  winsize   = Duration of the time window to process the video (in seconds)
#  winsizeGT = Duration of the time window to process the ground truth signal (in seconds)
#  timeStep  = Time step of the estimation (in seconds)
#  methods   = A list of methods to test (['CHROM','Green','ICA','LGI','PBV','PCA','POS','SSR'])
#
## Video signal Preprocessing
#
#  zeroMeanSTDnorm = Apply Zero Mean and Unit Standard Deviation (0/1)
#  detrending      = Apply detrenting algorithm (0/1)
#  detrMethod      = Detrenting algorithm (tarvainen/scipy)
#  detLambda       = If detrending = 1, regularization parameter of detrending algorithm
#  BPfilter        = Apply band pass filtering (0/1)
#  minHz           = If BPfilter = 1, the lower cut-off frequency (in hertz)
#  maxHz           = If BPfilter = 1, the upper cut-off frequency (in hertz)

[DEFAULT]
winSize         = 5
winSizeGT       = 5
timeStep        = 1
methods         = ['POS','CHROM']
zeroMeanSTDnorm = 0
detrending      = 0
detLambda       = 10
BPfilter        = 1
minHz           = 0.75
maxHz           = 4.0

## Video signal
#
#  dataset      = Name of the dataset to test ('PURE', 'UBFC1', 'UBFC2', 'LGI-PPGI', 'COHFACE', 'MAHNOB')
#  videoIdx     = A list of IDs reffered to the videos to test (eg. [0,1,2,...]) 
#                 or the string 'all' to test on the whole database
#  detector     = Method used for face detection (mtcnn, dlib, mtcnn_kalman)
#  extractor    = Preferred library to read video files (opencv/skvideo)
#  startTime    = Process video file from  start time (in seconds)
#  endTime      = Process video file until end time (in seconds). If < 0: process until (video length - endTime) 

[VIDEO]
dataset     = SAMPLE
videodataDIR= ../SAMPLE/
BVPdataDIR  = ../SAMPLE/
videoIdx    = [0]
detector    = mtcnn
extractor   = skvideo
startTime   = 3
endTime     = -3
ROImask = skin_fix
skinFix   = [40, 60]
skinAdapt = 0.2
rectCoords= [[0, 0, 150, 150]]
evm = 0
stat = mean

## Method specific configurations

[CHROM]
zeroMeanSTDnorm = 0
detrending      = 1
detrMethod      = scipy
BPfilter        = 0

[POS]
zeroMeanSTDnorm = 0
detrending      = 0
BPfilter        = 0

When I try to create my dataset with this code:

# -- dataset object
dataset = datasetFactory("SAMPLE", videodataDIR='../SAMPLE', BVPdataDIR='../SAMPLE')

I get this error:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
~/connor_asi/rppg_tests/pyVHR/pyVHR/datasets/dataset.py in datasetFactory(datasetName, videodataDIR, BVPdataDIR)
     10         datasetModule = import_module('pyVHR.datasets.' + moduleName) #, package='pyVHR')
---> 11         classOBJ = getattr(datasetModule, className)
     12         #obj = classOBJ(*args, **kwargs)

AttributeError: module 'pyVHR.datasets.sample' has no attribute 'SAMPLE'

During handling of the above exception, another exception occurred:

ImportError                               Traceback (most recent call last)
<ipython-input-223-01cfb362e158> in <module>
      1 # -- dataset object
----> 3 dataset = datasetFactory("SAMPLE", videodataDIR='../SAMPLE', BVPdataDIR='../SAMPLE')
      4 
      5 # -- videos filenames of the dataset

~/connor_asi/rppg_tests/pyVHR/pyVHR/datasets/dataset.py in datasetFactory(datasetName, videodataDIR, BVPdataDIR)
     14 
     15     except (AttributeError, ModuleNotFoundError):
---> 16         raise ImportError('{} is not part of pyVHR dataset collection!'.format(datasetName))
     17 
     18     return obj

ImportError: SAMPLE is not part of pyVHR dataset collection!

Any ideas what is going wrong? I can't find any other places I need to instantiate my custom data and I believe I've followed your examples precisely.

Thank you!

[vcuculo]

@connormeaton sorry for the late reply, but we were unable to reproduce your error. The configuration seems ok. Did you try with the basic_demo applied on one video before creating a new dataset? Does the latest release v0.0.4 eventually solve your problem?

[vcuculo]

Closing this issue since related to a previous version of pyVHR.