Implement Exchange.readtree and Exchange.writetree

[Lawrence37]

Exchange.readtree and Exchange.writetree will be used in place of shutil.copytree. readtree recursively reads files (not including ignored files) from a source directory and converts it to an encoded directory tree. writetree writes files (not including ignored files) from an encoded directory tree to the destination directory.

[Lawrence37]

Here's an (untested) example for writetree. src here is not exactly an encoded directory tree because the file contents have already been converted to binary.

def writetree(self, src, dest, ignore):
    """
    A custom implementation of shutil.copytree for saving files downloaded
    from the server.

    Arguments
    ---------
    src: list of dictionaries with keys 'path' and 'content'
        The source from which to copy. The path is relative to the current
        directory. The file content is a byte object.
    dest: str
        The destination directory path.
    ignore: function
        A function that returns True if the file should be ignored, False
        otherwise. The function takes as arguments the file directory path,
        file name, and file size in KB. Similar to the function returned by
        shutil.ignore_patterns.
    """

    for src_file in src:
        src_filename = src_file['path']
        dirname = os.path.split(src_filename)[0]
        filename = os.path.split(src_filename)[1]
        filesize = len(src_file['content'])
        if ignore(dirname, filename, filesize):
            continue
        dest_filename = os.path.join(dest, src_filename)
        with open(dest_filename, 'wb') as dest_file:
            dest_file.write(src_file['content'])

Here's a function that returns another function suitable for the ignore parameter.

def ignore_patterns(exclude=None, include=None, max_file_size=None, log=None):
    def ignore_patterns(directory, filename, filesize):
        rationale = None
        fullname = os.path.join(directory, filename)
        if exclude and any(fnmatch.fnmatch(filename, glob) for glob in exclude):
            if log:
                log.debug("Ignoring excluded file '{}' (see config option CourseDirectory.ignore)".format(fullname))
            return True
        elif include and not any(fnmatch.fnmatch(filename, glob) for glob in include):
            if log:
                log.debug("Ignoring non included file '{}' (see config option CourseDirectory.include)".format(fullname))
            return True
        elif max_file_size and filesize > 1000*max_file_size:
            if log:
                log.warning("Ignoring file too large '{}' (see config option CourseDirectory.max_file_size)".format(fullname))
            return True
    return ignore_patterns

[lxylxy123456]

@Lawrence37 Is this related to front end or backend? You may want to transfer this issue to nbgrader repo.

[Lawrence37]

I am aware I posted this on the wrong repo. I don't/didn't want to move it to avoid further confusion. You should ignore this issue.

[lxylxy123456]

So can I close it?

[Lawrence37]

Don't close it. Just ignore.

[lxylxy123456]

PS: it seems that I have to change some settings to allow issues in https://github.com/lxylxy123456/nbgrader. Now you can post issues at https://github.com/lxylxy123456/nbgrader/issues.

[aalmanza1998]

closing this issue. I implemented encode_dir and decode_dir in the base Exchange class to have this functionality.