Here we present simple tools for common bioinformatic use-cases, as well as other convenient functionality for your everyday command-line hacker. The main functionality takes in a file and a specified type, and will validate the file plus perform any simple stats, permutations, or filters. We were tired of getting errors when shoving data that wasn't first validated into pipelines, as well as performing the same bash and python hacks to look a little deeper at our large files. Further, we wanted a way to extend file validation and hacks to custom file types in our lab. Lastly, we wanted a more robust way to configure settings and record our behavior so we could be more reproducible without expending any extra effort.
This is a Python package, requiring the version of Python to be 3.6 or later (a higher number than 3.6). From your Terminal (mac) or terminal-emulator in Windows (putty, powershell, etc.), which we will refer to as your terminal, check if you have Python3 installed.
python --version
or
python3 --version
One of those commands should show "Python 3.6" or newer if you have python3 installed correctly. For the remaining steps, use the command (python or python3) that worked.
Installation should be pretty quick using the package manager pip. We recommend first creating and entering a virtual environment before installing our package, but this is not necessary. Perform the following command to install the package:
pip install bioinformatics_tools
One of the first things you may notice about this library is the interesting way in which you invoke it. Invocations always start with an executable from our library, such as:
fileflux
and they must always have a type: EXAMPLE key-value pair, as each executable is specific to the type of file it receives.
If you want to know what sort of arguments you're able to run for a particular executable, you just need to pass the word help as the first positional argument (the first word after fileflux), along with specifying the type:
fileflux help type: fasta
The type: fasta follows the pattern where you have a key followed by a colon (key:), then provide the name of a value separated by a space, e.g., key: value.
Above, this will first provide a status report that consists of 4 lines, which look something like this:
# Success
## Status
200: okay
## Response
Line one will tell you if the command succeeded or failed, lines 2 and 3 will give you a more specific status code, and line 4 will provide the response, which is often the part of the command output you'll be interested in.
After, it will show a list of every possible command, along with the docstring content (or help message) for each said command. These list of commands will start with 0 and go up through the last command (sorted alphabetically). Notice if you change the type the possible commands will change.
To run an executable, you must first type the executable's name (e.g., fileflux) and immediately after provide positional arguments that align with the action you wish to perform. For example, from the fileflux help page, you see:
$ fileflux seq length type: fasta file: example.fasta
Let's break down this command.
fileflux is the executable and is always required first.
seq length is a series of 2 positional arguments that conform to an executable found in the help message. These positional arguments are what tell the executable what to do, and may be anywhere from 1 to 5 space-separated arguments.
type: TYPE (in this case, type: fasta) is always required. This tells the executable what type of file we are dealing with. In later releases of this app, we will try to smart-parse files so this is not necessarily needed.
file: FILE (in this case, file: example.fasta) is also always required for any action to occur. This is the file you want to perform some action on.
Great, but what about if I want to provide extra parameters to the command? You can specify as many parameters as you want using the following syntax:
key: value
which will specify the parameter you want and the value it will have. For example:
length: 100 width: 200
will tell our system that we have the parameter length=100 and width=200.
For the following command
fileflux filter sequences type: fasta file: example.fasta length: 20
Note that the position of the parameters, the type, and the file do not matter. The above command is the same as the below command:
fileflux filter sequences length: 20 type: fasta file: example.fasta
The only positional arguments that matter are the fileflux as the first and the next sequence of arguments specifying the action to invoke (filter sequences)
One of our least favorite things to do is specify the same parameters over and over again, or forget which parameters we used because we did not document concisely enough. By default, we check your current directory for a file named config-caragols.yaml. Attributes put into these files will automatically be added to the configuration, which is available for all your programs to use.
For example, say you have a file config-caragols.yaml with the following parameters
name: "dane"
length: 10If you were to invoke the following command:
fileflux filter sequences type: fasta file: example.fasta
and the action filter sequences took in a length parameter, the action can grab that length parameter from the configuration file by default. Now let's say you have that configuration file but it does not have the length parameter specified. No worries, because each action will have default parameters baked into it. If you wanted to change the parameter on the fly without having to edit your config-caragols.yaml file, you can easily do that by specifying the parameter on the command line:
fileflux filter sequences type: fasta file: example.fasta length: 20
If you have no configuration file in your current directory, the "default config" will be used, which exists inside this tool. See Advanced Users to learn more about this.
If you have a config file that lives somewhere else outside your current directory, you can pass the path to file as part of the command, using the --config-file flag
fileflux filter sequences type: fasta file: example.fasta --config-file /tmp/myconfig.yamlIn summary, all the following commands are equivalent
fileflux filter sequences type: fasta file: example.fasta length: 20 - uses the "default config"
fileflux filter sequences type: fasta file: example.fasta - where the "local config", config-caragols.yaml, is located in the current directory, and contains length: 20
fileflux filter sequences type: fasta file: example.fasta --config-file /tmp/conf.yaml - where /tmp/conf.yaml contains length: 20
For those trying to share configs across multiple users (for example, a bioinformatician lab manager), you can modify the default configuration file. To see the default configuration info, run
python -m bioinformatics_tools.caragols.configuratorThe maintenance-info section is a starter guide to help track configuration files used in shared environments. It is not used by the application yet, so you can replace it or modify it in any way you want that makes sense for you. The configuration file content is logged each time a command is ran, so it might be useful for debugging with users to add information here. Future versions of bioinformatics_tools may rely on such a section to detect when a local config is out of date with the default config, that way users can keep up with the recommended configuration for the lab even when they have defined their own configuration files.
Logs from each time you run a command are saved to your hard drive. By default in ~/.caragols. These logs may be useful to look back on if you forget some work you did, or when you experience bugs, and us developers need more information in order to help. Up to 200MB of logs will be stored, after which, the oldest logs will start to get deleted.
You can configure some locations and settings for these logs. To update, edit the file at path reported after running this command
python -c 'import bioinformatics_tools.caragols.logger; print(bioinformatics_tools.caragols.logger.CONFIG_PATH)'The contents of the file should look something like this
{
"logging": {
"console_log_level": "INFO",
"directory": "~/.caragols/logs",
"use_user_subdir": true
}
}console_log_level: change this to WARNING if you want less information put
directory: is where logs will be stored on your machine
user_user_subdir: if true, the logs will be put one folder deeper (than that defined by directory), in a folder named after the current user. For example, if the user bobbyboucher runs any commands, their logs will appear under ~/.caragols/logs/bobbyboucher
Often times we get a file in our hands of a known type, but...how confident are we that the file exactly adheres to the type specification? For example, we might encounter a fasta file that has a V in the sequence, which may not be a big deal, but shoving the file into a pipeline may error out because a particular program determines the fasta file to be invalid. Or perhaps there's a pesky extra space in a file and another program was tricked into thinking the remaining file was empty and the program terminates early; we probably never would know that our program did not finish correctly! (This has happened to me before).
So here's our solution, which is meant to be an open-source, collaborative effort. We are creating validation classes that scrub our files squeeky clean (and add additional logging) before performing any work on them.
There's a folder called FileTypes, which contains Python classes that validate as many bioinformatics file types as possible. Each described file has the following capabilities and qualities:
$ fileflux valid type: fasta file: example.py # Return True or False$ fileflux write confident type: fasta file: exampleThis project adheres to semantic versioning, and we are in our 0.y.z phase, so this project is not signifying initial development. These releases are not considered stable, and their APIs can change frequently and without warning.
Dane Deemer: Senior Computational Biologist (Purdue University) Nathan Denny: Lead Research Analyst (Purdue University) Stephen Lindemann: Associate Professor (Purdue University) Maverick Cook: Senior Software Engineer (Purdue University)