Passing Videos to Topographica.

[Hima-Mehta]

What could be the possible/best way to pass videos as training patterns in Topographica? One way could be converting them into frames (not sure how do I pass that array of frames in sequence) . Any other way possible?

[jbednar]

What we have done before is to split a video into frames using ffmpeg, then use the Image pattern type as shown on the ImaGen homepage.

That example uses only a single image, but you can see that it's in a list, and you can replace that list with a list comprehension that iterates over all the frames by using a format string that matches whatever you've named your frames. E.g.:

>>> ["frame%05d.png"%f for f in range(5)]
['frame00000.png', 'frame00001.png', 'frame00002.png', 'frame00003.png', 'frame00004.png']

It wouldn't be difficult to write a PatternGenerator class that used a video library to render frames "on the fly", akin to how the OpenCV live camera support in topographica/topo/hardware/opencvcamera.py works, but using separate frames has been sufficient for what we've needed to do so far.

[Hima-Mehta]

But ig.Selector will select those frames randomly right; whereas in videos those frames are sequential.

[mjabri]

I wrote below some time ago which i still use. Maybe not most efficient, but it works for me. See doc string.

class CNFileSequenceImageGenerator(PatternGenerator): """ PatternGenerator of image sequences. List of lists (sequences of images). that produces images from files (e.g. gen_type: NumpyFile). Note the list can be either a stack or circular. If a stack, sequences are popped and list is drained. if circular, the index self.index_list points to the sequence that will be fetched next. Note the sequence in the list like a stack, elements are popped. When stack is empty, a new sequence is popped. The parameters of the files is supplied in parameter List "generators". generators: List of parameters to use as argument for the instance class that produces the image. The elements of the list are: (gen_type, filename, scale, orientation, cache_flag) size: Scaling factor applied to all images. circular: Boolean flag. If true, restart from begining (wraps) when at end of list. """ generators = param.List(precedence=0.97,bounds=(0,None), default=[], doc="List of sequences of files.") size = param.Number(default=1.0,doc="Scaling factor applied to all sub-patterns.") circular = param.Boolean(default=False,doc="Indicates whether list is circular (wraps)") sequence_length=param.Integer(default=0,doc="""Length of sequences being severed.""")

index_list = -1
index_seq  = -1
current_pg = None
current_seq  = None
current_targets=[]
current_subjects_poses = []

def function(self,p):
    """
    Returns next pattern in the sequence if available, and if not pop/get a sequence.
    from the list, and if at end of list, then rewind the list and pop/get a sequence.
    """

    pg = None
    need_to_get_seq = False
    el = None

    # We only pop from lists is not circular. Otherwise we get element using an index for both
    # element/sequence from generator and from sequence.
    # Note we do not rewind sequences.

    if self.current_seq is not None:
        # get/pop an element from the sequence if possible
        if p.circular:
            if self.index_seq >= len(self.current_seq) or self.index_seq < 0:
                need_to_get_seq = True
            else:
                el = self.current_seq[self.index_seq]
                self.index_seq += 1
        else:
            try:
                el = self.current_seq.pop(0) # may generate exception which is ok.
            except IndexError:
                # No more, so signal need new sequence
                need_to_get_seq = True
    else:
        # Have not popped out a sequence from list yet
        need_to_get_seq = True

    if need_to_get_seq:
        # Get a sequence from the list
        if p.circular:
            # do we need to rewind?
            if self.index_list >= len(p.generators) or self.index_list < 0:
                self.index_list = 0
            self.current_seq = p.generators[self.index_list]
            self.index_list += 1
            self.index_seq = 0
        else:
            try:
                self.current_seq = p.generators.pop(0)
            except IndexError:
                raise StopIteration

        # I am here so means i got a new sequence
        # pop/get an element from the sequence
        if p.circular:
            el = self.current_seq[self.index_seq]
            self.index_seq += 1
        else:
            try:
                el = self.current_seq.pop(0)
            except IndexError:
                raise StopIteration

    (image_reader, filename, parameters, rect, subject_id, pose_id, target) = el

    size = parameters.picture_scale if hasattr(parameters, 'picture_scale') else 1.0 # yes, overwrite default
    orientation = parameters.orientation if hasattr(parameters, 'orientation') else 0.0
    cache_image = parameters.cache_image if hasattr(parameters, 'cache_image') else False
    pg = image_reader(filename=filename, size = size, orientation=orientation,
                      cache_image=cache_image)

    # save pg
    if self.current_pg != None:
        del self.current_pg
    self.current_pg = pg

    # Push target on current_targets FIFO
    self.current_targets.insert(0, target) # insert in front
    if len(self.current_targets) > p.sequence_length:
        self.current_targets.pop() # pop oldest

    # save subject_id/pose_id in current_subjects_poses FIFO
    self.current_subjects_poses.insert(0, (subject_id, pose_id)) # insert in front
    if len(self.current_subjects_poses) > p.sequence_length:
        self.current_subjects_poses.pop() # pop oldest

    if pg is not None:
        return pg(xdensity=p.xdensity,ydensity=p.ydensity,bounds=p.bounds,
                     x=p.x+p.size*(pg.x*cos(p.orientation)-pg.y*sin(p.orientation)),
                     y=p.y+p.size*(pg.x*sin(p.orientation)+pg.y*cos(p.orientation)),
                     orientation=pg.orientation+p.orientation,size=pg.size*p.size,
                     scale=pg.scale*p.scale,offset=pg.offset+p.offset)

def get_current_generator(self):
    """Return the current generator."""
    if self.current_pg == None:
        error('Generator has not applied any patterns patterns yet.')
        raise IndexError
    return self.current_pg

def reset(self):
    """Reset generator to beginning. Only meaningful for circular generator."""
    self.index_list = -1
    self.index_seq  = -1
    if self.current_pg != None:
        del self.current_pg
        self.current_pg = None
    self.current_seq  = None

[jbednar]

But ig.Selector will select those frames randomly right; whereas in videos those frames are sequential.

No, the order of selection in ig.Selector is fully configurable, and is controlled by the ig.index parameter. By default, it is set to a uniform random value from 0 to 1, but you can set that to whatever you like to get whatever order you prefer. E.g. for an increasing order, you can set it to ng.ScaledTime(factor=X), where X determines how often a new frame will be selected.

You'll also want to set cache_image=False for the FileImage object, because you'll never be returning to the same image again.

[Hima-Mehta]

Thanks for your reply.I don't think my topographica is upgraded and have ScaledTime included in Numbergen. Is there any way to Upgrade it. Simple replacing New Init files for Imagegen,param and numbergen didn't work !

[jbednar]

I'm not sure what version you're using, so I can't say how to go about changing it, but if you use the current Git version it should work fine. If you are not ready to upgrade, you could copy the definition of ScaledTime to your own files; it's quite trivial:

class ScaledTime(NumberGenerator, TimeDependent):
    """The current time multiplied by some conversion factor."""

    factor = param.Number(default=1.0, doc="""
       The factor to be multiplied by the current time value.""")

    def __call__(self):
        return float(self.time_fn() * self.factor)

The surrounding functions might have changed since whatever you're using, but whatever they might have been at that time it should be straightforward to have an object that returns a scaled version of the current time.