Installation

It is advised to familiar yourself with the basic concepts of 101worker https://github.com/101companies/101worker/blob/master/README.md

The following installation guide assumes the usage of Ubuntu Linux. OSX works similar. Windows might work to some extent.

Install on Ubuntu Linux

git clone https://github.com/101companies/101worker.git
cd 101worker
make init -B
sudo make install-pip-pkgs -B
make download

Install on Windows

Install python3. Be sure to check "Add to path" when the installer prompts.

git clone https://github.com/101companies/101worker.git
cd 101worker
make init
make install-pip-pkgs
make download

Environment Variables are:

OMIT_GITDEPS - Skips pulling gitdeps
FULL_SWEEP - Perform a full sweep, disables incremental features
OMIT_PULL - Skips pulling 101repo

The full list of modules is configured https://github.com/101companies/101worker/blob/master/bin/worker

Development

Running a module

Use bin/run_module [moduleName] to run a single Module. The Test-Environment is specified in config/test_folders.txt which lists a number of directories from 101results/101repo to be copied to 101test/. This mode sets OMIT_GITDEPS, FULL_SWEEP and OMIT_PULL. This also deletes any dumps created by the module. Note for this to work the behavior must be configured. For more verbose output please use -v [2 or 3].

(Windows users have to use python bin/run_module [moduleName])

Resource Derivation

An example module can be found at https://github.com/101companies/101worker/tree/master/modules/simpleLOC. It counts the lines of code of every source file in 101repo.

Dump Collection

An example module for this can be found at https://github.com/101companies/101worker/tree/master/modules/locPerContribution. This modules collects all Loc files and puts them into a per contribution dump.

Third party Resource

An example module for this can be found at https://github.com/101companies/101worker/tree/master/modules/extractFacts. It takes a resource generated by another module to extract facts about the source code.

Wiki Dump

An example module for this can be found at https://github.com/101companies/101worker/tree/master/modules/moretagclouds. It parses the wiki dump and produces some tag clouds. These kinds of modules are not incremental.

Modules

Structure

Modules define the following names:

run(context, c) or run(context) depending on the configuration (see below).
test()
config

config

A dictionary which describes the required contract for the module. Keys are:

wantdiff - gets called for every incremental change
wantsfiles - not incremental but needs all files from 101repo (like FULL_SWEEP), if set to false and wantdiff is false as well, the module run is only run(context)
threadsafe - not used atm
behavior - defines input and output (optional)

behavior

config = {
    'behavior': {
        'creates': [['dump', 'locPerContribution']],
        'uses': [['resource', 'loc']]
    }
}

Note that creates and uses are lists of lists.

run(context, c)

Define this method if you use wantdiff or wantsfiles. c has the following Structure:

{
  'type': 'NEW_FILE|FILE_CHANGED|DELETED_FILE',
  'file': 'contributions/some-contribution/some-file.py'
}

Use this for modules which operate on single files, also for modules which work on derived resources.

run(context)

Define this method for modules which set wantdiff and wantsfiles to false. Use this for modules which use dumps or external resources.

test()

This method is used for testing, run bin/test to execute all tests for every module. Executing the tests executes the test function for every module. The test function should be in most cases:

def test():
    suite = unittest.TestLoader().loadTestsFromTestCase(ExampleModuleTest)
    unittest.TextTestRunner(verbosity=2).run(suite)

Refer to simpleLoc for an example. Testing the run function requires some kind of mocking usually, refer to the python documentation for further details. Note that you should make sure that the necessary functions are called with the correct parameters, use assert_called_with.

101companies / 101worker

readme