.. image:: https://github.com/arcanaframework/fileformats/actions/workflows/ci-cd.yml/badge.svg :target: https://github.com/arcanaframework/fileformats/actions/workflows/ci-cd.yml .. image:: https://codecov.io/gh/arcanaframework/fileformats/branch/main/graph/badge.svg?token=UIS0OGPST7 :target: https://codecov.io/gh/arcanaframework/fileformats .. image:: https://img.shields.io/pypi/pyversions/fileformats.svg :target: https://pypi.python.org/pypi/fileformats/ :alt: Supported Python versions .. image:: https://img.shields.io/pypi/v/fileformats.svg :target: https://pypi.python.org/pypi/fileformats/ :alt: Latest Version .. image:: https://img.shields.io/badge/docs-latest-brightgreen.svg?style=flat :target: https://arcanaframework.github.io/fileformats/ :alt: Documentation Status
Fileformats provides a library of file-format types implemented as Python classes.
The file-format types were designed to be used in type validation and data movement
during the construction and execution of data workflows. However, they can can also be
used some basic data handling methods (e.g. loading data to dictionaries) and format
conversions between some equivalent types via methods defined in the associated fileformats-extras <https://pypi.org/project/fileformats-extras/>
__
package.
File-format types are typically identified by a combination of file extension and "magic numbers" where applicable. However, unlike many other file-type Python packages, FileFormats, supports multi-file data formats ("file sets") often found in scientific workflows, e.g. with separate header/data files. FileFormats also provides a flexible framework to add custom identification routines for exotic file formats, e.g. formats that require inspection of headers to locate data files, directories containing certain file types, or to peek at metadata fields to define specific sub-types (e.g. functional MRI DICOM file set). It is in the handling of multi-file formats that fileformats comes into its own, since it keeps track of auxiliary files when moving/copying to different file-system locations and calculating hashes.
See the extension template <https://github.com/ArcanaFramework/fileformats-extension-template>
for instructions on how to design FileFormats extensions modules to augment the
standard file-types implemented in the main repository with custom domain/vendor-specific
file-format types (e.g. fileformats-medimage <https://pypi.org/project/fileformats-medimage/>
).
Support for all non-vendor standard MIME types (i.e. ones not matching */vnd.*
or */x-*
) has been
added to FileFormats by semi-automatically scraping the IANA MIME types
_ website for file
extensions and magic numbers. As such, many of the formats in the library have not been properly
tested on real data and so should be treated with some caution. If you encounter any issues with an implemented file
type, please raise an issue in the GitHub tracker <https://github.com/ArcanaFramework/fileformats/issues>
__.
Adding support for vendor formats will be relatively straightforward and is planned for v1.0.
FileFormats can be installed for Python >= 3.7 from PyPI with
.. code-block:: bash
$ python3 -m pip fileformats
Support for converter methods between a few select formats can be installed by passing the 'extras' package, e.g
.. code-block:: bash
$ python3 -m pip install fileformats-extras
Using the WithMagicNumber
mixin class, the Png
format can be defined concisely as
.. code-block:: python
from fileformats.generic import File
from fileformats.core.mixin import WithMagicNumber
class Png(WithMagicNumber, File):
binary = True
ext = ".png"
iana_mime = "image/png"
magic_number = b".PNG"
Files can then be checked to see whether they are of PNG format by
.. code-block:: python
png = Png("/path/to/image/file.png") # Checks the extension and magic number
which will raise a FormatMismatchError
if initialisation or validation fails, or
for a boolean method that checks the validation use matches
.. code-block:: python
if Png.matches(a_path_to_a_file):
... handle case ...
There are 2 main functions that can be used for format identification
fileformats.core.from_mime
fileformats.core.find_matching
from_mime
As the name suggests, this function is used to return the FileFormats class corresponding to a given `MIME <https://www.iana.org/assignments/media-types/media-types.xhtml>`__ string. All non-vendor official MIME-types are supported. Non-official types can be loaded using the `application/x-name-of-type`
form as long as the name of the type is unique amongst all installed format types. To avoid name clashes between different extension types, the "MIME-like" string can be used instead, where informal registries corresponding to the fileformats extension namespace are used instead, e.g. `medimage/nifti-gz` or `datascience/hdf5`.
``find_matching``
Given a set of file-system paths, by default, find_matching
will iterate through all installed fileformats classes and return all that validate successfully (formats without any specific constraints are excluded by default). The potential candidate classes can be restricted by using the candidates
keyword argument.
While not implemented in the main File-formats itself, file-formats provides hooks for other packages to implement extra behaviour such as format conversion. The fileformats-extras <https://github.com/ArcanaFramework/fileformats-extras>
__ implements a number of converters between standard file-format types, e.g. archive types to/from generic file/directories, which if installed can be called using the convert()
method.
.. code-block:: python
from fileformats.application import Zip
from fileformats.generic import Directory
zip_file = Zip.convert(Directory("/path/to/a/directory"))
extracted = Directory.convert(zip_file)
copied = extracted.copy_to("/path/to/output")
The converters are implemented in the Pydra dataflow framework, and can be linked into wider Pydra workflows by creating a converter task
.. code-block:: python
import pydra
from pydra.tasks.mypackage import MyTask
from fileformats.application import Json, Yaml
wf = pydra.Workflow(name="a_workflow", input_spec=["in_json"])
wf.add(
Yaml.get_converter(Json, name="json2yaml", in_file=wf.lzin.in_json)
)
wf.add(
MyTask(
name="my_task",
in_file=wf.json2yaml.lzout.out_file,
)
)
...
Alternatively, the conversion can be executed outside of a Pydra_ workflow with
.. code-block:: python
json_file = Json("/path/to/file.json")
yaml_file = Yaml.convert(json_file)
This work is licensed under a
Creative Commons Attribution 4.0 International License <http://creativecommons.org/licenses/by/4.0/>
_
.. image:: https://i.creativecommons.org/l/by/4.0/88x31.png :target: http://creativecommons.org/licenses/by/4.0/ :alt: Creative Commons Attribution 4.0 International License
.. _Pydra: https://pydra.readthedocs.io
.. Fastr: https://gitlab.com/radiology/infrastructure/fastr
.. IANA MIME types
: https://www.iana_mime.org/assignments/media-types/media-types.xhtml