Set up running of new SWIP code through SWAMP

[von]

• uploaded pegasus-worker-4.8.0dev-x86_64_rhel_7.tar.gz (Python 2.x scripts)
• Assessed using all Python tools: Pylint, Flake8, Bandit
• Successfully completed (sorta), but haven’t analyzed.

[von]

The goal here is that new code developed for SWIP will be "assured" using the SWAMP, not the pre-existing Pegasus corpus. Can we create a snapshot of Pegasus' current weaknesses and then make a "diff" going forward (what are the semantically meaningful changes).

[von]

There will be other Pegasus changes concurrent with SWIP and we may not have the effort to run all of those through the SWAMP. Do we do the SWIP changes in a branch (that is merged often) and just run that branch through the SWAMP? (This is primarily a @rynge and @vahi question.)

[rheiland]

For starters, we've run an initial 'diff' between SWAMP (scarf, xml) results returned from 'beta3' and 4.8.0: https://docs.google.com/document/d/1sQmKMvIjaTbL-LkLDdFijj8hnG0QkkmLnp76zW-akSM/edit#

We'll continue to improve this "diff" process (scripts, etc) to generate useful output.

[rheiland]

New approach: use github repos to feed SWAMP and use some continuous integration (CI) tool, e.g. TravisCI, to do whatever "diff" we end up writing. Unknown if we can extract just a portion (subdir) from a github repo. Will explore. Will also explore options for CI.

[rheiland]

I've been emailing SWAMP Support to try to understand their assessments (and reporting) better. The latest volley: _Can someone explain and walk me through a simple github assessment on SWAMP? I’ve created https://github.com/rheiland/test_swamp1 and created a release https://github.com/rheiland/test_swamp1/releases . I’d like to perform an assessment on this release. Note that it contains two subdirs, one containing a Python script; the other, two C files (neither one compilable into an executable)._

_I login to my SWAMP account using my username/password (not via github account). I then ‘Add New Package’ and select ‘Remote Git repo’. For ‘External URL’, I provide https://github.com/rheiland/test_swamp1/archive/v1.0.tar.gz and provide no ‘Checkout args’. On the next screen, I leave ‘Package path’ alone, i.e. “./“ and select C/C++ as Language. Next screen: I leave it as ‘No build’. Continue… Run Assessment, select a Tool, e.g. cppcheck, select a Platform. Run Assessment. Fails._ ...

I've also asked them (again) about their contradictory reporting of failure/success of an assessment on web site vs. email.

[rheiland]

More email exchanges w/ SWAMP Support - trying to understand how I can automate, as much as possible, SWAMP and Pegasus releases in GitHub, e.g.: Kyle,

Thanks for the reply. No, I didn’t expect SWAMP to do >1 language per run. But I’m still confused with the workflow and my resulting error. I specify a URL to a release of my test repo to upload it to SWAMP: https://github.com/rheiland/test_swamp1/archive/v1.0.tar.gz

On the next screen, I don’t know what “Package path" I should supply. My gzipped tarball untars into:

/test_swamp1-1.0
     README.md
     /c_code
     /python3_code

So I tried entering “./python3_code” for my Package path and proceed with the assessment, but it fails with the error “Unpacking archive ‘v1.0.tar’ failed".

Hoping you can provide some insight. I've attached some, hopefully, relevant screen captures. -Randy

[rheiland]

getVuln.txt While I wait for SWAMP Support to reply, I'm experimenting with Python to parse and provide relevant info from the SWAMP SCARF results, with two goals in mind: 1) provide a more helpful (concise, etc) listing of vulnerabilities from the assessment, 2) lead up to an automated "diff" of SCARF files.

$ python getVuln.py 

---- scarf-Bandit-4.8beta3-orig.xml:
AnalyzerReport
{'platform_name': 'ubuntu-16.04-64', 'build_root_dir': '/home/builder/build', 'package_version': '4.8.0beta3', 'parser_fw': 'resultparser', 'assessment_start_ts': '1503595449.7361057', 'parser_fw_version': '3.1.1', 'uuid': '6bd923a0-89e4-4e12-a8a3-922dbd9f0d42', 'tool_name': 'bandit', 'package_root_dir': 'pkg1', 'package_name': 'pegasus-python-source', 'tool_version': '1.3.0'}

---- HIGH priority ----
jinja2_autoescape_false HIGH
subprocess_popen_with_shell_equals_true HIGH
subprocess_popen_with_shell_equals_true HIGH
subprocess_popen_with_shell_equals_true HIGH
subprocess_popen_with_shell_equals_true HIGH
subprocess_popen_with_shell_equals_true HIGH
subprocess_popen_with_shell_equals_true HIGH
subprocess_popen_with_shell_equals_true HIGH
subprocess_popen_with_shell_equals_true HIGH

---- MEDIUM priority ----
hardcoded_tmp_directory MEDIUM
hardcoded_tmp_directory MEDIUM
blacklist MEDIUM

[rheiland]

Got reply from SWAMP Support - it's not possible to specify a URL to a github Release. Rather, just a URL to the repo and, optionally, a branch, etc.. Also, the SWAMP expects files to have a suffix to indicate their type, e.g., ".py". Exchanged emails on swip-l about the reasoning for missing suffixes on Pegasus scripts in /bin.

Here's the reply from SWAMP Support (note large discrepancy in # of weaknesses between tools for "pegasus-integrity" Python script):

When creating a SWAMP package from a GitHub repo, SWAMP does a git clone with the "External URL" provided and then does a git checkout with the "Checkout Argument" provided. It then creates a compressed archive with the results and submits it as a package to the SWAMP, the same as if a user had uploaded an archive. If no "Checkout Argument" is provided, the package will contain the latest code in the master branch.

To get the "External URL" from a GitHub repo, click the green "Clone or download" button on the repo's main page. In this case that would be: https://github.com/rheiland/test_swamp1.git

The "Check out argument", which is optional, would be the name of a branch, release, tag, or a commit number. In this case that would be the release name: v1.0

I tried this Creating a new package in the SWAMP with the following parameters:

"External URL" = https://github.com/rheiland/test_swamp1.git "Check out argument" = v1.0 "Package path" = python3_code/ "Language" = Python "Python Version" = Python 3 "Build System" = No build

I was able to successfully create this package and run assessments.

However, no files were actually assessed. I got back zero weaknesses from all three python tools. This is because the python code is in a file in the Git Repo without any extension: "python3_code/pegasus-integrity" without the file extension, SWAMP does not know that this file contains python code and does not assess it. That is also why, when adding the package, on the source page, the SWAMP does not recognize that the package could be a python package.

I downloaded the pegasus-integrity file to my local system and renamed it "pegasus-integrity.py" and zipped it. I then created a Python 3 no build package in the SWAMP uploading that file.

The SWAMP recognized that it was a python package.

I was able to assess it and get results: Flake8 found 117 potential weaknesses; Pylint found 1. I suggest adding a file extension to the "pegasus-integrity" file in the GitHub repo and creating a new release.

[rheiland]

Significant updates to the getVuln.py mentioned above, but now called parseSCARF.py. Added it and a sample SCARF .xml file to a "/static_analysis" subdirectory in the github repo. And did a minor revision to the README.md

[rheiland]

Partial summary: SWAMP:

• cannot perform more than one language assessment at once
• files need suffixes (e.g., ".py") to determine what type(s) of assessment tools are possible
• user cannot programmatically download SCARF (SWAMP Common Assessment Result Format) files
• "diff" of SCARF results will target differences between vulnerabilities (found by same assessment tool, but of different versions of files); rf. https://github.com/IU-CACR/SWIP/blob/master/static_analysis/sample_output.txt

Pegasus:

• unclear if/how we will assess SWIP-only edits to repo

[rheiland]

Recently looked into using SWAMP's REST API for programmatic functionality to perform assessments and retrieve results (SCARF output; SCARF=SWAMP Common Assessment Result Format). Unfortunately, the REST API does not currently allow retrieving SCARFs. SWAMP Support suggested I try their Java-CLI. Still a work in progress, but the Java-CLI does indeed let us retrieve SCARFs, e.g.:

$ alias jcli='java -jar ./java-cli-1.3.5-jar-with-dependencies.jar'
$ jcli   # get list of cmds
$ jcli projects --list
$ jlci assess --list -P b77ede58-bb9d-11e4-a4cd-001a4a814425
UUID                                  Package Name    Package Version Tool Name       Tool Version    Platform Name   Platform Version
04ee42ff-3a49-4e6e-8cac-46a63321029d  test_swamp1     latest          Pylint          latest          default         latest
89d15494-096f-4ea8-99c2-00fa501d8303  test_swamp1     latest          Flake8          latest          default         latest
ebb174cc-c5f0-4740-899f-6bb628fd4dac  test_swamp1     latest          Bandit          latest          default         latest
...

$ jcli status -P b77ede58-bb9d-11e4-a4cd-001a4a814425 -A 89d15494-096f-4ea8-99c2-00fa501d8303
SUCCESS, 117, ac65d12c-0a97-11e8-9925-001a4a814413
   ## means found 117 weaknesses
$ java -jar ./java-cli-1.3.5-jar-with-dependencies.jar results -P b77ede58-bb9d-11e4-a4cd-001a4a814425 -R ac65d12c-0a97-11e8-9925-001a4a814413 -F /Users/heiland/swamp.txt
$ file ~/swamp.txt
/Users/heiland/swamp.txt: XML 1.0 document text, ASCII text, with very long lines

[rheiland]

At the March AHM, it was decided that we want to run all of the Pegasus code through SWAMP, not just SWIP-related code. Additionally, it was decided to focus on just code Releases, not dev code in the github repo. Given this, we will approach SWAMP from two directions:

1) upload Python scripts, e.g., from /bin (rename them to have .py suffix): .../bin$ grep python *|grep usr pegasus-analyzer:#!/usr/bin/env python pegasus-dagman:#!/usr/bin/env python pegasus-db-admin:#!/usr/bin/env python pegasus-em:#!/usr/bin/env python pegasus-exitcode:#!/usr/bin/env python pegasus-globus-online:#!/usr/bin/env python pegasus-globus-online-init:#!/usr/bin/env python pegasus-graphviz:#!/usr/bin/env python pegasus-init:#!/usr/bin/env python pegasus-integrity:#!/usr/bin/env python pegasus-metadata:#!/usr/bin/env python2.7 pegasus-monitord:#!/usr/bin/env python pegasus-plots:#!/usr/bin/env python pegasus-s3:#!/usr/bin/env python pegasus-service:#!/usr/bin/env python pegasus-statistics:#!/usr/bin/env python pegasus-submitdir:#!/usr/bin/env python pegasus-transfer:#!/usr/bin/env python

and perform assessments on those using various Python static analysis tools, and

2) upload .java code and perform assessments on those using various Java static analysis tools, and

3) upload .c code and perform assessments on those using various C static analysis tools.

Note that SWAMP assumes both Java and C code is compilable. We will attempt to use the Java-CLI to do the upload, assessment, and retrieval of results.

[rheiland]

Mats provided a pegasus-swamp-*-4.9.0dev tarball that appends a .py to the Python scripts in /bin (and other places?, e.g. /test)

Recall some Java-CLI cmds:

$ alias jcli='java -jar ./java-cli-1.3.5-jar-with-dependencies.jar'
$ jcli 
------------------------------------------------------------------------
Usage:   
------------------------------------------------------------------------
 must be one of the following:
        login
        logout
        assess
        project OR projects
        package OR packages
        platforms OR platforms
        results
        status
        tool OR tools
        user
------------------------------------------------------------------------
For information on the  for a  execute:
      --help or   -H

$ jcli login --help
usage: Command Line Parameters
 -C,--console                           Accepts username and password from
                                        the terminal
 -F,--filepath    Properties file containing
                                        username and password
 -H,--help                              Shows Help
 -Q,--quiet                             Less verbose output
 -S,--swamp-host            URL for SWAMP host: default is
                                        https://www.mir-swamp.org

$ jcli login --console
USERNAME: heiland
PASSWORD: 

I go to the SWAMP Web UI and create a new project for SWIP (since the Java-CLI can't seem to do that), then:

$ jcli project --list

UUID                                  Create Date                    Name                 
07a07fd3-3b68-40bc-89ff-a09cf8d74212  'Mon Mar 12 20:36:31 EDT 2018' Scientific Workflow Integrity with Pegasus
7c54f385-1c91-12ab-8d23-0ef4961ff000  'Fri Mar 07 14:21:50 EST 2014' Science Gateways Platform as a service
43efc9a7-09ff-b50d-d3b5-1acce4b751bf  'Tue Aug 26 09:08:01 EDT 2014' Quiz                 
b77ede58-bb9d-11e4-a4cd-001a4a814425  'Mon Feb 23 20:51:19 EST 2015' MyProject

[rheiland]

Using one of Mats' new "swamp" tarballs at https://download.pegasus.isi.edu/pegasus/4.9.0dev, I tested doing an assessment of the /bin scripts to see if SWAMP could successfully assess just the Python scripts and not choke on the non-Python scripts. It can. But I had to properly specify the "Path" to be "." since I built the tarball in-place in /bin; SWAMP wanted to specify the path to be "pegasus-", apparently because my tarball was named pegasus_bin_4.9.0dev.tar.gz.

[rheiland]

Running our Python script to parse one of the SCARF files: $ python parseSCARF.py scarf-Bandit.xml 1 1 0 flags = 1 1 0

---- scarf-Bandit.xml: AnalyzerReport {'tool_name': 'bandit', 'platform_name': 'ubuntu-16.04-64', 'package_version': 'unknown', 'assessment_start_ts': '1522088324.7147691', 'uuid': '3d076660-4a05-49cc-aec7-331a209c49d9', 'parser_fw_version': '3.1.4', 'package_name': 'pegasus_bin_4.9.0dev', 'tool_version': '1.3.0', 'parser_fw': 'resultparser', 'build_root_dir': '/home/builder/build', 'package_root_dir': 'pkg1'}

================== High priority Line 27 in pkg1/pegasus-db-admin.py ==> subprocess call with shell=True identified, security issue. Line 29 in pkg1/pegasus-db-admin.py ==> subprocess call with shell=True identified, security issue. Line 9 in pkg1/pegasus-em.py ==> subprocess call with shell=True identified, security issue. Line 11 in pkg1/pegasus-em.py ==> subprocess call with shell=True identified, security issue. Line 32 in pkg1/pegasus-exitcode.py ==> subprocess call with shell=True identified, security issue. Line 11 in pkg1/pegasus-globus-online-init.py ==> subprocess call with shell=True identified, security issue. Line 13 in pkg1/pegasus-globus-online-init.py ==> subprocess call with shell=True identified, security issue. Line 9 in pkg1/pegasus-service.py ==> subprocess call with shell=True identified, security issue. Line 11 in pkg1/pegasus-service.py ==> subprocess call with shell=True identified, security issue. Line 9 in pkg1/pegasus-submitdir.py ==> subprocess call with shell=True identified, security issue. Line 11 in pkg1/pegasus-submitdir.py ==> subprocess call with shell=True identified, security issue.

----- Got 11 high priority vulnerabilities.

================== Medium priority

----- Got 0 medium priority vulnerabilities.

[rheiland]

Next steps:

• Grab earlier Pegasus version, append ".py" to /bin Python scripts, and run same assessments. Compare results of parseSCARF in various ways, e.g. raw count of High & Med priority vulns.
• Determine steps to have SWAMP assess just the C code in the tarball.
• See if we can do all this via the Java CLI.

[rheiland]

Updated to https://github.com/mirswamp/java-cli/releases/tag/releases%2F1.4.2 after a vuln was patched: "Fixes a vulnerability that allows an attacker with an account on the same host to impersonate the user’s SWAMP identity and gain access their SWAMP account. For each successful attack, the attacker will be able to impersonate the user for a maximum time period of two days."

Yes, of course, a vuln in a tool that helps us prevent vulns. It's all just software after all.

[rheiland]

Downloaded pegasus-4.7.4, renamed /bin scripts to have .py (and .pl) suffixes, uploaded to SWAMP, ran assessment (from web interface), downloaded SCARF files.

~/Documents/Von/SWIP/pegasus-4.7.4/bin$ python parseSCARF.py scarf_Bandit.xml 1 1 0 flags = 1 1 0

---- scarf_Bandit.xml: AnalyzerReport {'package_name': 'peg_bin4.7.4', 'uuid': '80e891e1-c8d1-4850-b094-0b4e38ba8a85', 'build_root_dir': '/home/builder/build', 'parser_fw_version': '3.1.4', 'package_version': 'unknown', 'tool_name': 'bandit', 'parser_fw': 'resultparser', 'package_root_dir': 'pkg1', 'assessment_start_ts': '1523289924.1410673', 'tool_version': '1.3.0', 'platform_name': 'ubuntu-16.04-64'}

================== High priority Line 27 in pkg1/pegasus-db-admin.py ==> subprocess call with shell=True identified, security issue. Line 29 in pkg1/pegasus-db-admin.py ==> subprocess call with shell=True identified, security issue. Line 9 in pkg1/pegasus-em.py ==> subprocess call with shell=True identified, security issue. Line 11 in pkg1/pegasus-em.py ==> subprocess call with shell=True identified, security issue. Line 32 in pkg1/pegasus-exitcode.py ==> subprocess call with shell=True identified, security issue. Line 9 in pkg1/pegasus-service.py ==> subprocess call with shell=True identified, security issue. Line 11 in pkg1/pegasus-service.py ==> subprocess call with shell=True identified, security issue. Line 9 in pkg1/pegasus-submitdir.py ==> subprocess call with shell=True identified, security issue. Line 11 in pkg1/pegasus-submitdir.py ==> subprocess call with shell=True identified, security issue. ----- Got 9 high priority vulnerabilities.

================== Medium priority ----- Got 0 medium priority vulnerabilities.

[rheiland]

Remember when we ran the assessments on the newer 4.9.0dev earlier, we had: ----- Got 11 high priority vulnerabilities. vs. the 9 here for v4.7.4.

Here, a simple 'diff' of the text files reporting "high vulns" between versions produces meaningful results:

.../pegasus-swamp-python-4.9.0dev/bin$ diff scarf_Bandit_high.txt ../../pegasus-4.7.4/bin/scarf_Bandit_high.txt 
7,8d6
< Line 11 in pkg1/pegasus-globus-online-init.py ==> subprocess call with shell=True identified, security issue.
< Line 13 in pkg1/pegasus-globus-online-init.py ==> subprocess call with shell=True identified, security issue.
14c12
< ----- Got 11 high priority vulnerabilities.
---
> ----- Got 9 high priority vulnerabilities.

Somewhat ironically, the newer version of this particular script (with 2 more high vulns) is shorter:

.../pegasus-swamp-python-4.9.0dev/bin$ wc -l pegasus-globus-online.py 
     355 pegasus-globus-online.py
.../pegasus-4.7.4/bin$ wc -l pegasus-globus-online.py 
     391 pegasus-globus-online.py

But that won't always be the case, of course. It will depend on how much files have changed, among other things.

[rheiland]

Compare SCARF results of the Pylint assessment tool:

.../pegasus-4.7.4/bin$ python parseSCARF.py scarf_Pylint.xml 1 1 0
flags =  1 1 0

---- scarf_Pylint.xml:
AnalyzerReport
{'tool_name': 'PyLint', 'assessment_start_ts': '1523289919.387744', 'build_root_dir': '/home/builder/build', 'package_root_dir': 'pkg1', 'tool_version': '1.6.4', 'parser_fw': 'resultparser', 'package_version': 'unknown', 'uuid': '593da995-9cdc-4f86-8923-c774d80997a4', 'platform_name': 'ubuntu-16.04-64', 'parser_fw_version': '3.1.4', 'package_name': 'peg_bin4.7.4'}

================== High priority
----- Got  0  high priority vulnerabilities.

================== Medium priority
Line 53   in  pkg1/pegasus-globus-online.py  ==>  invalid syntax
Line 779  in  pkg1/pegasus-transfer.py  ==>  invalid syntax
Line 47   in  pkg1/pegasus-monitord.py  ==>  invalid syntax
Line 94   in  pkg1/pegasus-plots.py  ==>  inconsistent use of tabs and spaces in indentation
Line 17   in  pkg1/pegasus-em.py  ==>  Unable to import 'Pegasus.service.ensembles.commands'
Line 17   in  pkg1/pegasus-service.py  ==>  Unable to import 'Pegasus.service.server'
Line 9    in  pkg1/pegasus-gridftp.py  ==>  invalid syntax
Line 351  in  pkg1/pegasus-statistics.py  ==>  Missing parentheses in call to 'print'
Line 9    in  pkg1/pegasus-init.py  ==>  invalid syntax
Line 14   in  pkg1/pegasus-s3.py  ==>  Unable to import 'Pegasus'
Line 84   in  pkg1/pegasus-metadata.py  ==>  Missing parentheses in call to 'print'
Line 35   in  pkg1/pegasus-exitcode.py  ==>  Unable to import 'Pegasus'
Line 193  in  pkg1/pegasus-graphviz.py  ==>  invalid syntax
Line 17   in  pkg1/pegasus-submitdir.py  ==>  Unable to import 'Pegasus.submitdir'
Line 338  in  pkg1/pegasus-analyzer.py  ==>  invalid token
Line 61   in  pkg1/pegasus-dagman.py  ==>  Missing parentheses in call to 'print'
Line 35   in  pkg1/pegasus-db-admin.py  ==>  Unable to import 'Pegasus.db.admin'

----- Got  17  medium priority vulnerabilities.

---

vs. newer, v4.9.0dev version:

~/Documents/Von/SWIP/pegasus-swamp-python-4.9.0dev/bin$ !py
python parseSCARF.py scarf-Pylint.xml 1 1 0
flags =  1 1 0

---- scarf-Pylint.xml:
AnalyzerReport
{'parser_fw_version': '3.1.4', 'tool_name': 'PyLint', 'package_root_dir': 'pkg1', 'platform_name': 'ubuntu-16.04-64', 'build_root_dir': '/home/builder/build', 'package_version': 'unknown', 'uuid': 'f75a42d9-af65-40b7-a18f-214b7c2f3534', 'assessment_start_ts': '1522088348.0925484', 'parser_fw': 'resultparser', 'tool_version': '1.6.4', 'package_name': 'pegasus_bin_4.9.0dev'}

================== High priority
----- Got  0  high priority vulnerabilities.

================== Medium priority
Line 17   in  pkg1/pegasus-service.py  ==>  Unable to import 'Pegasus.service.server'
Line 263  in  pkg1/pegasus-globus-online.py  ==>  invalid syntax
Line 35   in  pkg1/pegasus-exitcode.py  ==>  Unable to import 'Pegasus'
Line 17   in  pkg1/pegasus-submitdir.py  ==>  Unable to import 'Pegasus.submitdir'
Line 35   in  pkg1/pegasus-db-admin.py  ==>  Unable to import 'Pegasus.db.admin'
Line 94   in  pkg1/pegasus-plots.py  ==>  inconsistent use of tabs and spaces in indentation
Line 338  in  pkg1/pegasus-analyzer.py  ==>  invalid token
Line 47   in  pkg1/pegasus-monitord.py  ==>  invalid syntax
Line 14   in  pkg1/pegasus-s3.py  ==>  Unable to import 'Pegasus'
Line 17   in  pkg1/pegasus-em.py  ==>  Unable to import 'Pegasus.service.ensembles.commands'
Line 84   in  pkg1/pegasus-metadata.py  ==>  Missing parentheses in call to 'print'
Line 351  in  pkg1/pegasus-statistics.py  ==>  Missing parentheses in call to 'print'
Line 738  in  pkg1/pegasus-transfer.py  ==>  invalid syntax
Line 61   in  pkg1/pegasus-dagman.py  ==>  Missing parentheses in call to 'print'
Line 193  in  pkg1/pegasus-graphviz.py  ==>  invalid syntax
Line 355  in  pkg1/pegasus-integrity.py  ==>  invalid syntax
Line 9    in  pkg1/pegasus-init.py  ==>  invalid syntax
Line 6    in  pkg1/pegasus-globus-online-init.py  ==>  Unable to import 'ConfigParser'
Line 19   in  pkg1/pegasus-globus-online-init.py  ==>  Unable to import 'globus_sdk'

----- Got  19  medium priority vulnerabilities.

[rheiland]

Note to self re all past SWAMP tickets: https://ticket.continuousassurance.org/ticket/list/my

eScience draft: https://www.overleaf.com/15521189zcfcwqsqqysv#/58904178/

[rheiland]

~/Documents/Von/SWIP/pegasus-4.7.0/bin$ python parseSCARF.py scarf-bandit.xml 1 1 0
flags =  1 1 0

---- scarf-bandit.xml:
AnalyzerReport
{'package_version': 'unknown', 'package_name': 'pegasus_4.7.0_bin', 'package_root_dir': 'pkg1', 'tool_name': 'bandit', 'platform_name': 'ubuntu-16.04-64', 'uuid': '0c920188-b44f-486c-869a-b5fb44500b73', 'parser_fw_version': '3.1.4', 'assessment_start_ts': '1523904214.3629181', 'tool_version': '1.3.0', 'parser_fw': 'resultparser', 'build_root_dir': '/home/builder/build'}

================== High priority
Line 27   in  pkg1/bin/pegasus-db-admin.py  ==>  subprocess call with shell=True identified, security issue.
Line 29   in  pkg1/bin/pegasus-db-admin.py  ==>  subprocess call with shell=True identified, security issue.
Line 9    in  pkg1/bin/pegasus-em.py  ==>  subprocess call with shell=True identified, security issue.
Line 11   in  pkg1/bin/pegasus-em.py  ==>  subprocess call with shell=True identified, security issue.
Line 32   in  pkg1/bin/pegasus-exitcode.py  ==>  subprocess call with shell=True identified, security issue.
Line 9    in  pkg1/bin/pegasus-service.py  ==>  subprocess call with shell=True identified, security issue.
Line 11   in  pkg1/bin/pegasus-service.py  ==>  subprocess call with shell=True identified, security issue.
Line 9    in  pkg1/bin/pegasus-submitdir.py  ==>  subprocess call with shell=True identified, security issue.
Line 11   in  pkg1/bin/pegasus-submitdir.py  ==>  subprocess call with shell=True identified, security issue.

----- Got  9  high priority vulnerabilities.

================== Medium priority

----- Got  0  medium priority vulnerabilities.

[rheiland]

~/Documents/Von/SWIP/pegasus-4.7.0/bin$ python parseSCARF.py scarf-pylint.xml 1 1 0
flags =  1 1 0

---- scarf-pylint.xml:
AnalyzerReport
{'parser_fw': 'resultparser', 'package_root_dir': 'pkg1', 'parser_fw_version': '3.1.4', 'uuid': '3a1a0c01-a5e3-4be3-ad96-b4f5e523464e', 'assessment_start_ts': '1523904213.9353237', 'tool_version': '1.6.4', 'package_name': 'pegasus_4.7.0_bin', 'build_root_dir': '/home/builder/build', 'tool_name': 'PyLint', 'platform_name': 'ubuntu-16.04-64', 'package_version': 'unknown'}

================== High priority

----- Got  0  high priority vulnerabilities.

================== Medium priority
Line 14   in  pkg1/bin/pegasus-s3.py  ==>  Unable to import 'Pegasus'
Line 351  in  pkg1/bin/pegasus-statistics.py  ==>  Missing parentheses in call to 'print'
Line 53   in  pkg1/bin/pegasus-globus-online.py  ==>  invalid syntax
Line 47   in  pkg1/bin/pegasus-monitord.py  ==>  invalid syntax
Line 94   in  pkg1/bin/pegasus-plots.py  ==>  inconsistent use of tabs and spaces in indentation
Line 35   in  pkg1/bin/pegasus-exitcode.py  ==>  Unable to import 'Pegasus'
Line 9    in  pkg1/bin/pegasus-gridftp.py  ==>  invalid syntax
Line 17   in  pkg1/bin/pegasus-service.py  ==>  Unable to import 'Pegasus.service.server'
Line 84   in  pkg1/bin/pegasus-metadata.py  ==>  Missing parentheses in call to 'print'
Line 748  in  pkg1/bin/pegasus-transfer.py  ==>  invalid syntax
Line 17   in  pkg1/bin/pegasus-em.py  ==>  Unable to import 'Pegasus.service.ensembles.commands'
Line 193  in  pkg1/bin/pegasus-graphviz.py  ==>  invalid syntax
Line 17   in  pkg1/bin/pegasus-submitdir.py  ==>  Unable to import 'Pegasus.submitdir'
Line 35   in  pkg1/bin/pegasus-db-admin.py  ==>  Unable to import 'Pegasus.db.admin'
Line 61   in  pkg1/bin/pegasus-dagman.py  ==>  Missing parentheses in call to 'print'
Line 9    in  pkg1/bin/pegasus-init.py  ==>  invalid syntax
Line 338  in  pkg1/bin/pegasus-analyzer.py  ==>  invalid token

----- Got  17  medium priority vulnerabilities.

[rheiland]

The Flake8 Python assessment tool has some pretty ridiculous "medium priority vulns" reporting:

~/Documents/Von/SWIP/pegasus-4.7.0/bin$ python parseSCARF.py scarf-flake8.xml 1 1 0
flags =  1 1 0

---- scarf-flake8.xml:
AnalyzerReport
{'parser_fw': 'resultparser', 'parser_fw_version': '3.1.4', 'package_root_dir': 'pkg1', 'package_name': 'pegasus_4.7.0_bin', 'build_root_dir': '/home/builder/build', 'package_version': 'unknown', 'assessment_start_ts': '1523904217.1960688', 'tool_name': 'flake8', 'platform_name': 'ubuntu-16.04-64', 'uuid': '77e0d381-fa0b-4dc3-911e-ddcdd0528e5b', 'tool_version': '3.2.1'}

================== High priority

----- Got  0  high priority vulnerabilities.

================== Medium priority
Line 35   in  pkg1/bin/pegasus-metadata.py  ==>   line too long (83 > 79 characters)
Line 36   in  pkg1/bin/pegasus-metadata.py  ==>   line too long (99 > 79 characters)
Line 37   in  pkg1/bin/pegasus-metadata.py  ==>   line too long (93 > 79 characters)
Line 38   in  pkg1/bin/pegasus-metadata.py  ==>   line too long (103 > 79 characters)
Line 44   in  pkg1/bin/pegasus-metadata.py  ==>   expected 2 blank lines after class or function definition, found 1
Line 46   in  pkg1/bin/pegasus-metadata.py  ==>   module level import not at top of file
...
----- Got  2237  medium priority vulnerabilities.

[rheiland]

Trying to bundle/upload some Pegasus C/C++ code for analysis:

~/Documents/Von/SWIP/pegasus-4.7.0/src$ tar cf - tools >tools.tar
then gzip, upload to SWAMP, tell it to use the Makefile there... fails:

FILE: build/build_stderr.out from output/build.tar.gz
/bin/sh: /home/builder/build/pkg1/tools/pegasus-kickstart/../../../release-tools/getversion: No such file or directory
pegasus-kickstart.c: In function ‘main’:
pegasus-kickstart.c:425:22: error: ‘PEGASUS_VERSION’ undeclared (first use in this function)
                 puts(PEGASUS_VERSION);
                      ^
pegasus-kickstart.c:425:22: note: each undeclared identifier is reported only once for each function it appears in
make[1]: *** [pegasus-kickstart.o] Error 1

Obviously, I need to upload more of the Pegasus repo, possibly everything, and provide more specific directions telling SWAMP how to compile sections.

[rheiland]

First attempt at full-blown assessment of entire pegasus-4.7.0 repo on SWAMP. Five tools available for Java/ant build: OWASP Dependency Check, checkstyle, SpotBugs, PMD, and error-prone. All 5 failed. I've emailed support at SWAMP for help in understanding the failures, as they're not obvious to me.

[rheiland]

Uploading source from here: https://github.com/pegasus-isi/pegasus/releases/tag/4.7.0 and editing the build.xml to prevent an error related to checking a git hash: /Users/heiland/dev/pegasus-4.7.0/build.xml Comment out the following ‘exec’ block in the build.xml:

   exec executable="/bin/bash" outputproperty="pegasus.build.git.hash" failonerror="true"
    ...

     [exec] machine/linux.c:293:9: warning: ignoring return value of â<80><98>fscanfâ<80><99>, declared with attribute warn_unused_result [-Wunused-result]
     [exec]          fscanf(f, "%lf %lf", &uptime, idletime);
     [exec]          ^
     [exec] gcc  -Wall -O2 -ggdb -std=gnu99  -DLINUX syscall.c -c -o syscall.o
     [exec] gcc  getif.o utils.o useinfo.o statinfo.o jobinfo.o limitinfo.o machine.o machine/basic.o appinfo.o parse.o mysystem.o mylist.o invoke.o pegasus-kickstart.o procinfo.o machine/linux.o syscall.o -lm -o pegasus-kickstart
     [exec] Makefile:66: recipe for target 'pegasus-kickstart' failed
     [exec] make[1]: Leaving directory '/home/builder/build/pkg1/pegasus-4.7.0/src/tools/pegasus-kickstart'
     [exec] Makefile:11: recipe for target 'install' failed
     [exec] getif.o: file not recognized: File format not recognized
     [exec] collect2: error: ld returned 1 exit status
     [exec] make[1]: *** [pegasus-kickstart] Error 1
     [exec] make: *** [install] Error 1

---

Received a reply from Vamshi (SWAMP support) about this and he claimed he was successful by using the following tarball; however, he didn't provide enough details. I attempted to use the same tarball, but still had problems.

Uploading source tarball from here: http://download.pegasus.isi.edu/pegasus/4.7.0/ results in:

...
compile-python:
    [mkdir] Created dir: /home/builder/build/pkg1/pegasus-source-4.7.0/dist/pegasus-4.7.0/lib/python2.7/dist-packages
     [exec] Traceback (most recent call last):
     [exec]   File "setup.py", line 4, in 
     [exec]     from setuptools import setup, find_packages
     [exec] ImportError: No module named setuptools

I attempted to indicate 'setuptools' was a (Python) dependency in the build, but obviously, it didn't work.

https://jira.cosalab.org:8443/servicedesk/customer/portal/8/SH-78

Have sent emails to service@continuousassurance.org, but no replies today, not even automated.

[rheiland]

To edit the "Build" info associated with an existing package:

On the "pegasus-source-4.7.0”, In the table displaying (Version, Notes, Date Added), click on the link for the version that you like to edit. This takes you to the "package version" screen. Here, choose the "Build" tab and click "Edit Build Info" button.

[rheiland]

Review both: https://docs.google.com/document/d/1sQmKMvIjaTbL-LkLDdFijj8hnG0QkkmLnp76zW-akSM/edit# https://github.com/IU-CACR/SWIP/tree/master/static_analysis

Mini-summary:

• cannot run SWAMP on multiple languages in Pegasus repo
• when attempting to use SWAMP to assess (all of) Pegasus, it will assess Java code
• need to extract non-Java code (Python, C) and assess separately

[rheiland]

The SCARF files will contain SWAMP paths to the files being assessed. These paths may differ between assessments, requiring manual string substitution (via "sed", etc) in order to do a meaningful Unix "diff". For example:

~/Documents/Von/SWIP/pegasus-4.7.0/bin$ head scarf-bandit-orig.xml (well, if GitHub Issues allowed verbatim text, I could show this XML!)

~/Documents/Von/SWIP/pegasus-4.7.0/bin$ sed -i -- 's/pkg1\/bin\///' scarf-bandit.xml 

---

Added more to: https://docs.google.com/document/d/1L2ZZp2u343d1-CeGq4dsvzept0fFQvetmwzN-onSlQM/edit#heading=h.6xucn9k6dk1s

[rheiland]

Possibly useful info (not related to SWAMP, but for [Pegasus] code in GitHub repo):

https://help.github.com/articles/about-security-alerts-for-vulnerable-dependencies/ GitHub tracks public vulnerabilities in Ruby gems, NPM and Python packages on MITRE's Common Vulnerabilities and Exposures (CVE) List.

When GitHub receives a notification of a newly-announced vulnerability, we identify public repositories (and private repositories that have opted in to vulnerability detection) that use the affected version of the dependency. Then, we send security alerts to owners and people with admin access to affected repositories. You can also configure security alerts for additional people or teams working in organization-owned repositories.

https://blog.github.com/2018-07-12-security-vulnerability-alerts-for-python/

[rheiland]

Began experimenting with the SCARF "diff" script (Perl) from the SWAMP devs (Kupsch), after a meeting Von & I had with them on 8/31/18:

Diff SCARF files: https://github.com/mirswamp/swamp-scarf-diff Library to read and process SCARF files: https://github.com/mirswamp/swamp-sarif-io Library to create and upload packages, trigger assessments, and download SCARF results: https://github.com/mirswamp/java-cli

I also had to install an additional Perl module dependency.

Primary caveat of their script: won't work/will be confusing if line #s change between versions of code being diff'd.

Simple Pegasus code examples: ~/Documents/Von/SWIP/SWAMP_2018/diff_Jim/swamp-scarf-diff[/data]: $ perl swamp_scarf_diff.pl data/scarf-bandit-orig.xml data/copy.xml $ zip -r pegasus-db-admin.zip pegasus-db-admin.py Upload to SWAMP, perform analysis, download SCARF(s). $ python parseSCARF.py scarf-bandit-admin1.xml 1 1 1

[rheiland]

A fix(?) for some of Pegasus Python scripts that have the vuln "subprocess call with shell=True": https://security.openstack.org/guidelines/dg_avoid-shell-true.html

[rheiland]

Von would like to know "What % vulnerabilities in SWIP changes are caught by diff?"

• Pegasus devs don't tag changes as "SWIP" changes.
• We will do this across releases, starting with 4.7.0 which was the release at the start of SWIP (~Oct 2016). The current (Sep 2018) release is 4.8.4.
• We will need to separate out and bundle files by language since SWAMP only handles one language per assessment.
• Recall that there are multiple SWAMP tools per language that can generate their own SCARF files.

[rynge]

Regarding the shell=True flags - I don't see that being an easy change as most of the command lines are dictated by the for example transfer tool we are calling out to. I don't think shell=True as problem for us, as the user already have full command line access where we are running these commands, so there is no "chaining additional commands" concerns when we shell out.

[rheiland]

https://jira.isi.edu/browse/PM-1179 Supposedly an umbrella JIRA issue for Integrity checking in Pegasus. But currently all issues there are Closed or Resolved? And what about, e.g., https://jira.isi.edu/browse/PM-1305 ? To search all issues for "integrity": https://jira.isi.edu/browse/PM-252?jql=text%20~%20%22integrity%22

[rheiland]

$ diff2 scarf_470_bin_bandit.xml scarf_474_bin_bandit.xml >diff_bandit_470_474.out (where "diff2" is an alias to run the Perl "diff" script provided by SWAMP, at https://github.com/mirswamp/swamp-scarf-diff)

diff_bandit_470_474.out: --- scarf_470_bin_bandit.xml +++ scarf_474_bin_bandit.xml

+ 1 BugInstances categorized by BugGroup: undefined, BugCode: subprocess_without_shell_equals_true, SourceFile: pkg1/pegasus-s3.py, StartLine: 9
+ 1 BugInstances categorized by BugGroup: undefined, BugCode: subprocess_without_shell_equals_true, SourceFile: pkg1/pegasus-s3.py, StartLine: 8
+ 1 BugInstances categorized by BugGroup: undefined, BugCode: blacklist, SourceFile: pkg1/pegasus-exitcode.py, StartLine: 27
+ 1 BugInstances categorized by BugGroup: undefined, BugCode: blacklist, SourceFile: pkg1/pegasus-em.py, StartLine: 4
+ 1 BugInstances categorized by BugGroup: undefined, BugCode: blacklist, SourceFile: pkg1/pegasus-db-admin.py, StartLine: 21
+ 1 BugInstances categorized by BugGroup: undefined, BugCode: blacklist, SourceFile: pkg1/pegasus-service.py, StartLine: 4
+ 1 BugInstances categorized by BugGroup: undefined, BugCode: blacklist, SourceFile: pkg1/pegasus-s3.py, StartLine: 4
+ 1 BugInstances categorized by BugGroup: undefined, BugCode: blacklist, SourceFile: pkg1/pegasus-submitdir.py, StartLine: 4
+ 1 BugInstances categorized by BugGroup: undefined, BugCode: subprocess_popen_with_shell_equals_true, SourceFile: pkg1/pegasus-db-admin.py, StartLine: 29
+ 1 BugInstances categorized by BugGroup: undefined, BugCode: subprocess_popen_with_shell_equals_true, SourceFile: pkg1/pegasus-db-admin.py, StartLine: 27
+ 1 BugInstances categorized by BugGroup: undefined, BugCode: subprocess_popen_with_shell_equals_true, SourceFile: pkg1/pegasus-submitdir.py, StartLine: 9
+ 1 BugInstances categorized by BugGroup: undefined, BugCode: subprocess_popen_with_shell_equals_true, SourceFile: pkg1/pegasus-submitdir.py, StartLine: 11
+ 1 BugInstances categorized by BugGroup: undefined, BugCode: subprocess_popen_with_shell_equals_true, SourceFile: pkg1/pegasus-service.py, StartLine: 11
+ 1 BugInstances categorized by BugGroup: undefined, BugCode: subprocess_popen_with_shell_equals_true, SourceFile: pkg1/pegasus-service.py, StartLine: 9
+ 1 BugInstances categorized by BugGroup: undefined, BugCode: subprocess_popen_with_shell_equals_true, SourceFile: pkg1/pegasus-exitcode.py, StartLine: 32
+ 1 BugInstances categorized by BugGroup: undefined, BugCode: subprocess_popen_with_shell_equals_true, SourceFile: pkg1/pegasus-em.py, StartLine: 11
+ 1 BugInstances categorized by BugGroup: undefined, BugCode: subprocess_popen_with_shell_equals_true, SourceFile: pkg1/pegasus-em.py, StartLine: 9

- 1 BugInstances categorized by BugGroup: undefined, BugCode: subprocess_without_shell_equals_true, SourceFile: pkg1/bin/pegasus-s3.py, StartLine: 8
- 1 BugInstances categorized by BugGroup: undefined, BugCode: subprocess_without_shell_equals_true, SourceFile: pkg1/bin/pegasus-s3.py, StartLine: 9
- 1 BugInstances categorized by BugGroup: undefined, BugCode: blacklist, SourceFile: pkg1/bin/pegasus-submitdir.py, StartLine: 4
- 1 BugInstances categorized by BugGroup: undefined, BugCode: blacklist, SourceFile: pkg1/bin/pegasus-exitcode.py, StartLine: 27
- 1 BugInstances categorized by BugGroup: undefined, BugCode: blacklist, SourceFile: pkg1/bin/pegasus-s3.py, StartLine: 4
- 1 BugInstances categorized by BugGroup: undefined, BugCode: blacklist, SourceFile: pkg1/bin/pegasus-db-admin.py, StartLine: 21
- 1 BugInstances categorized by BugGroup: undefined, BugCode: blacklist, SourceFile: pkg1/bin/pegasus-em.py, StartLine: 4
- 1 BugInstances categorized by BugGroup: undefined, BugCode: blacklist, SourceFile: pkg1/bin/pegasus-service.py, StartLine: 4
- 1 BugInstances categorized by BugGroup: undefined, BugCode: subprocess_popen_with_shell_equals_true, SourceFile: pkg1/bin/pegasus-submitdir.py, StartLine: 11
- 1 BugInstances categorized by BugGroup: undefined, BugCode: subprocess_popen_with_shell_equals_true, SourceFile: pkg1/bin/pegasus-submitdir.py, StartLine: 9
- 1 BugInstances categorized by BugGroup: undefined, BugCode: subprocess_popen_with_shell_equals_true, SourceFile: pkg1/bin/pegasus-exitcode.py, StartLine: 32
- 1 BugInstances categorized by BugGroup: undefined, BugCode: subprocess_popen_with_shell_equals_true, SourceFile: pkg1/bin/pegasus-db-admin.py, StartLine: 27
- 1 BugInstances categorized by BugGroup: undefined, BugCode: subprocess_popen_with_shell_equals_true, SourceFile: pkg1/bin/pegasus-db-admin.py, StartLine: 29
- 1 BugInstances categorized by BugGroup: undefined, BugCode: subprocess_popen_with_shell_equals_true, SourceFile: pkg1/bin/pegasus-service.py, StartLine: 11
- 1 BugInstances categorized by BugGroup: undefined, BugCode: subprocess_popen_with_shell_equals_true, SourceFile: pkg1/bin/pegasus-service.py, StartLine: 9
- 1 BugInstances categorized by BugGroup: undefined, BugCode: subprocess_popen_with_shell_equals_true, SourceFile: pkg1/bin/pegasus-em.py, StartLine: 9
- 1 BugInstances categorized by BugGroup: undefined, BugCode: subprocess_popen_with_shell_equals_true, SourceFile: pkg1/bin/pegasus-em.py, StartLine: 11

Files are identical:

~/Documents/Von/SWIP/SWAMP_2018/diff_Jim/swamp-scarf-diff/bin_python$ diff pegasus-s3_470.py pegasus-s3_474.py

Yet, the Perl scarf-diff script reports otherwise:

+ 1 BugInstances categorized by BugGroup: undefined, BugCode: subprocess_without_shell_equals_true, SourceFile: pkg1/pegasus-s3.py, StartLine: 9
+ 1 BugInstances categorized by BugGroup: undefined, BugCode: subprocess_without_shell_equals_true, SourceFile: pkg1/pegasus-s3.py, StartLine: 8
…
- 1 BugInstances categorized by BugGroup: undefined, BugCode: subprocess_without_shell_equals_true, SourceFile: pkg1/bin/pegasus-s3.py, StartLine: 8
- 1 BugInstances categorized by BugGroup: undefined, BugCode: subprocess_without_shell_equals_true, SourceFile: pkg1/bin/pegasus-s3.py, StartLine: 9

I suspect it has to do with the different paths on the "SourceFile"s. I've emailed Jim Kupsch to ask.

[rheiland]

Using this nice argument resolves the problem!

$ diff2 --no-source_file scarf_470_bin_bandit.xml scarf_474_bin_bandit.xml --- scarf_470_bin_bandit.xml +++ scarf_474_bin_bandit.xml $

I.e., no differences! But keep in mind, this will be the case when there are the very same vulnerabilities reported in both versions, i.e., none have been fixed.

[rheiland]

Emailed support@continuousassurance.org: I feel that I’ve been down this road before. I’m trying to assess Java code that’s in Pegasus 4.8.0 (https://github.com/pegasus-isi/pegasus/releases/tag/4.8.0). So I upload the .tar.gz to SWAMP, set a build dependency as ‘python-setuptools’ -- otherwise, I get an error related to that:

 _[exec]     from setuptools import setup, find_packages
 [exec] ImportError: No module named setuptools_

and then attempt an assessment using ‘ant’ to build the code.

I’ve attached the resulting error. Thoughts? thanks, Randy

Report Generation Time Mon Oct 22 18:49:47 2018 Build Command 'ant -Dhttps.protocols=TLSv1.2 -lib /home/builder/scripts/build-monitors/swamp-ant-listener/target/swamp-ant-listener-1.0.jar -listener swamp.AntBuildListener -noinput -nouserlib -Dswamp.build.monitor.output=/home/builder/build/build_artifacts.xml' failed with exit-code '1'. See 'build_stdout.out', 'build_stderr.out' for errors Contents of status.out NOTE: begin PASS: install-os-dependencies 25.106456s PASS: tool-runtime-compatibility 0.001036s PASS: package-unarchive 1.078033s SKIP: configure 0.000006s FAIL: build 2.308731s +++++++++++++

Command 'ant -Dhttps.protocols=TLSv1.2 -lib /home/builder/scripts/build-monitors/swamp-ant-listener/target/swamp-ant-listener-1.0.jar -listener swamp.AntBuildListener -noinput -nouserlib -Dswamp.build.monitor.output=/home/builder/build/build_artifacts.xml' failed with exit-code '1'. See 'build_stdout.out', 'build_stderr.out' for errors +++++++++++++

PASS: build-archive 6.627926s FAIL: all 35.131166s NOTE: end Contents of stderr FILE: build/build_stderr.out from output/build.tar.gz OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=2048M; support was removed in 8.0

BUILD FAILED /home/builder/build/pkg1/pegasus-4.8.0/build.xml:133: exec returned: 128

Total time: 2 seconds Contents of stdout FILE: build/build_stdout.out from output/build.tar.gz Buildfile: /home/builder/build/pkg1/pegasus-4.8.0/build.xml [echo] Pegasus Version: 4.8.0 [echo] JAVA TARGET is 1.6 [echo] JAVA SOURCE is 1.6 [echo] Build timestamp: 20181022182027Z [echo] Architecture: x86_64_ubuntu_16 [echo] OSID: ubuntu16 [echo] Library subdir: lib [echo] Python: /usr/bin/python2.7 [echo] Python library subdir: lib/python2.7/dist-packages

dist-clean: [exec] for i in pegasus-cluster pegasus-keg pegasus-kickstart; do \ [exec] if [ -d $i ]; then \ [exec] make -C $i distclean || exit 1; \ [exec] fi; \ [exec] done [exec] make[1]: Entering directory '/home/builder/build/pkg1/pegasus-4.8.0/src/tools/pegasus-cluster' [exec] gcc -MM .c > depends.mk [exec] rm -f .o core core. check.? depends.mk [exec] rm -f pegasus-cluster try-cpus [exec] make[1]: Leaving directory '/home/builder/build/pkg1/pegasus-4.8.0/src/tools/pegasus-cluster' [exec] make[1]: Entering directory '/home/builder/build/pkg1/pegasus-4.8.0/src/tools/pegasus-keg' [exec] rm -f pegasus-keg.o core core. basic.o linux.o [exec] rm -f pegasus-keg [exec] make[1]: Leaving directory '/home/builder/build/pkg1/pegasus-4.8.0/src/tools/pegasus-keg' [exec] make[1]: Entering directory '/home/builder/build/pkg1/pegasus-4.8.0/src/tools/pegasus-kickstart' [exec] /home/builder/build/pkg1/pegasus-4.8.0/src/tools/pegasus-kickstart/../../../release-tools/getversion --header > /home/builder/build/pkg1/pegasus-4.8.0/src/tools/pegasus-kickstart/version.h [exec] gcc -MM getif.c utils.c useinfo.c statinfo.c jobinfo.c limitinfo.c machine.c machine/basic.c appinfo.c parse.c mysystem.c mylist.c invoke.c pegasus-kickstart.c procinfo.c checksum.c machine/linux.c syscall.c > depends.mk [exec] rm -f pegasus-kickstart libinterpose.so [exec] make[1]: Leaving directory '/home/builder/build/pkg1/pegasus-4.8.0/src/tools/pegasus-kickstart' [exec] Makefile:88: To build pegasus-mpi-cluster set CXX to the path to your MPI C++ compiler wrapper [exec] /home/builder/build/pkg1/pegasus-4.8.0/src/tools/pegasus-mpi-cluster/../../../release-tools/getversion --header > /home/builder/build/pkg1/pegasus-4.8.0/src/tools/pegasus-mpi-cluster/version.h [exec] g++ -MM .cpp > depends.mk [exec] Makefile:88: To build pegasus-mpi-cluster set CXX to the path to your MPI C++ compiler wrapper [exec] rm -f .o test-strlib test-dag test-log test-engine test-tools test-fdcache test-protocol test-scheduler version.h depends.mk [exec] rm -f pegasus-mpi-cluster [exec] rm -rf /home/builder/build/pkg1/pegasus-4.8.0/lib/pegasus/externals [exec] Makefile:39: WARNING: pg_config not found: skipping python postgresql library [exec] Makefile:44: WARNING: mysql_config not found: skipping python mysql library [exec] running clean [exec] 'build/lib.linux-x86_64-2.7' does not exist -- can't clean it [exec] 'build/bdist.linux-x86_64' does not exist -- can't clean it [exec] 'build/scripts-2.7' does not exist -- can't clean it

set-git-hash: [echo] Setting pegasus.build.git.hash

[rynge]

I think the "Git download" version is not working on those older releases. Instead, clone and checkout 4.8.0. That should make git.hash work.

[rheiland]

I did this: $ git clone --single-branch -b 4.8 https://github.com/pegasus-isi/pegasus.git pegasus-4.8.0-clone and then zipped up that code and uploaded to SWAMP. But I still get failed assessments, e.g.,

for the "error-prone" SWAMP tool, I get: compile-r: [mkdir] Created dir: /home/builder/build/pkg1/pegasus-4.8.0-clone/dist/pegasus-4.8.5dev/share/pegasus/r [exec] R is not available. Either install the R development packages, or disable this part of the build by setting PEGASUS_BUILD_R_MODULES=0 in your environment before executing ant.

for the SpotBugs tool, I get: FILE: build/build_stdout.out from output/build.tar.gz Buildfile: /home/builder/build/pkg1/pegasus-4.8.0/build.xml [echo] Pegasus Version: 4.8.0 [echo] JAVA TARGET is 1.6 [echo] JAVA SOURCE is 1.6 [echo] Build timestamp: 20181022182027Z [echo] Architecture: x86_64_ubuntu_16 [echo] OSID: ubuntu16 [echo] Library subdir: lib [echo] Python: /usr/bin/python2.7 [echo] Python library subdir: lib/python2.7/dist-packages

dist-clean: [exec] for i in pegasus-cluster pegasus-keg pegasus-kickstart; do \ [exec] if [ -d $i ]; then \ [exec] make -C $i distclean || exit 1; \ [exec] fi; \ [exec] done [exec] make[1]: Entering directory '/home/builder/build/pkg1/pegasus-4.8.0/src/tools/pegasus-cluster' [exec] gcc -MM .c > depends.mk [exec] rm -f .o core core. check.? depends.mk [exec] rm -f pegasus-cluster try-cpus [exec] make[1]: Leaving directory '/home/builder/build/pkg1/pegasus-4.8.0/src/tools/pegasus-cluster' [exec] make[1]: Entering directory '/home/builder/build/pkg1/pegasus-4.8.0/src/tools/pegasus-keg' [exec] rm -f pegasus-keg.o core core. basic.o linux.o [exec] rm -f pegasus-keg [exec] make[1]: Leaving directory '/home/builder/build/pkg1/pegasus-4.8.0/src/tools/pegasus-keg' [exec] make[1]: Entering directory '/home/builder/build/pkg1/pegasus-4.8.0/src/tools/pegasus-kickstart' [exec] /home/builder/build/pkg1/pegasus-4.8.0/src/tools/pegasus-kickstart/../../../release-tools/getversion --header > /home/builder/build/pkg1/pegasus-4.8.0/src/tools/pegasus-kickstart/version.h [exec] gcc -MM getif.c utils.c useinfo.c statinfo.c jobinfo.c limitinfo.c machine.c machine/basic.c appinfo.c parse.c mysystem.c mylist.c invoke.c pegasus-kickstart.c procinfo.c checksum.c machine/linux.c syscall.c > depends.mk [exec] rm -f pegasus-kickstart libinterpose.so [exec] make[1]: Leaving directory '/home/builder/build/pkg1/pegasus-4.8.0/src/tools/pegasus-kickstart' [exec] Makefile:88: To build pegasus-mpi-cluster set CXX to the path to your MPI C++ compiler wrapper [exec] /home/builder/build/pkg1/pegasus-4.8.0/src/tools/pegasus-mpi-cluster/../../../release-tools/getversion --header > /home/builder/build/pkg1/pegasus-4.8.0/src/tools/pegasus-mpi-cluster/version.h [exec] g++ -MM .cpp > depends.mk [exec] Makefile:88: To build pegasus-mpi-cluster set CXX to the path to your MPI C++ compiler wrapper [exec] rm -f .o test-strlib test-dag test-log test-engine test-tools test-fdcache test-protocol test-scheduler version.h depends.mk [exec] rm -f pegasus-mpi-cluster [exec] rm -rf /home/builder/build/pkg1/pegasus-4.8.0/lib/pegasus/externals [exec] Makefile:39: WARNING: pg_config not found: skipping python postgresql library [exec] Makefile:44: WARNING: mysql_config not found: skipping python mysql library [exec] running clean [exec] 'build/lib.linux-x86_64-2.7' does not exist -- can't clean it [exec] 'build/bdist.linux-x86_64' does not exist -- can't clean it [exec] 'build/scripts-2.7' does not exist -- can't clean it

set-git-hash: [echo] Setting pegasus.build.git.hash

[rynge]

For "error-prone" set PEGASUS_BUILD_R_MODULES=0 in your environment.

SpotBugs still seems like a git checkout issue - I can fix this in new builds, but not for the released 4.8.0. Either use our released tarball (https://download.pegasus.isi.edu/pegasus/4.8.0/pegasus-source-4.8.0.tar.gz) or the git clone/checkout mention before. Do get an error after set-git-hash? Another explanation would be git not being installed.

[rheiland]

Downloaded https://download.pegasus.isi.edu/pegasus/4.8.0/pegasus-source-4.8.0.tar.gz and attempted a SWAMP assessment on that. Got following for both SpotBugs and error-prone:

compile-r: [mkdir] Created dir: /home/builder/build/pkg1/pegasus-source-4.8.0/dist/pegasus-4.8.0/share/pegasus/r [exec] R is not available. Either install the R development packages, or disable this part of the build by setting PEGASUS_BUILD_R_MODULES=0 in your environment before executing ant.

Will attempt to tell your build script to skip R via the env var, as you suggested. It's just that it's non-trivial to do this simple thing in SWAMP. Seems odd that R not being present is a show-stopper. Why wouldn't your build script simply detect that's the case, maybe print an info/warning msg and carry on? Or maybe this isn't the real reason for the assessment crapping out.

[rynge]

The top level build is using Apache Ant which is not great at detecting and making runtime decisions. We felt that the R parts are important enough that we want user to "opt out" instead of just disabling it automatically. I will bring this up with the team again, but there has never been much interest in improving the configurability of the build.

You should be able to just prepend the env var when calling ant. For example:

PEGASUS_BUILD_R_MODULES=0 ant dist

[rheiland]

Yes, I know that's the usual way of prepending an env var to a build, but, you see, this is SWAMP...

On Tue, Oct 23, 2018 at 12:29 PM Mats Rynge notifications@github.com wrote:

The top level build is using Apache Ant which is not great at detecting and making runtime decisions. We felt that the R parts are important enough that we want user to "opt out" instead of just disabling it automatically. I will bring this up with the team again, but there has never been much interest in improving the configurability of the build.

You should be able to just prepend the env var when calling ant. For example:

PEGASUS_BUILD_R_MODULES=0 ant dist

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/IU-CACR/SWIP/issues/8#issuecomment-432318390, or mute the thread https://github.com/notifications/unsubscribe-auth/ABsVRSDikTogEYZ-WQxNTg7HcqDrlwZiks5un0PTgaJpZM4Ohh_v .

[rheiland]

James Kupsch replied to my emailed Q to 'support':

If your package is an ant-based build and you want set ant properties, then you can add the following to build-options field of the package to set P1 to p1val and P2 to p2val

-DP1=p1val -DP2=p2val

If you mean setting an operating system environment variable that only get used within ant (the environment variable isn't required in any of the exec's that ant uses) and your build.xml file contains the following to make environment variables for use in the ant file:

Then you can add the following to change the value ${env.V1} returns (but not the environment variable V1 in any exec'd programs:

-Denv.V1=V1val

If you also need V1 set as an environment variable in an exec'd program, then there is no built-in way to accomplish this and you will need to modify your build.xml file to add the following to each \<exec> entity:

\<env key="V1" value="${env.V1}"\/>

This will work if V1 is set as an environment variable or is passed to ant with -D.

Finally, if you just need to set the environment in the configure command, this uses bash shell syntax, so you can prefix the command with any environment variables to set.

V1=v1val V2=v2val ./your-configure-command arg1

[rheiland]

Trying to provide the env var: PEGASUS_BUILD_R_MODULES=0

---

still results in a build error:

[rheiland]

A different approach - still errors.

[rheiland]

Finally, successful Java assessments of source. But not until I hand-edit the build.xml (as I had to do before). So, now the procedure seems to be:

    <!-- Get Git Hash if it isn't already set -->
    <echo>Setting pegasus.build.git.hash</echo>
<!-- rwh
    <exec executable="/bin/bash" outputproperty="pegasus.build.git.hash" failonerror="true">
        <arg value="-c"/>
        <arg value="git rev-parse HEAD"/>
    </exec>
-->
  </target>
...
  <target name="compile-r" description="Compile R DAX API">
<!--  rwh
      <mkdir dir="${dist.share}/r"/>
      <exec executable="./setup.sh" dir="lib/pegasus/r/Pegasus" failonerror="true">
        <arg line="${dist.share}/r" />
      </exec>
-->
  </target>

  <target name="dist-r" depends="dist-clean,compile-r" description="Copy R DAX tarball to dist folder">
<!-- rwh
      <copy preservelastmodified="true" todir="dist">
          <fileset dir="${dist.share}/r" includes="*.tar.gz"/>
          <mapper>
            <mapper type="regexp" from="^(.*)\.tar\.gz" to="pegasus-r-\1.tar.gz"/>
          </mapper>
      </copy>
-->
  </target>

Still need to specify python-setuptools as a Package dependency in SWAMP.

[rheiland]

A gentle reminder of why the heck we're doing this... we want to use the new SWAMP-provided "SCARF diff" (Perl) tool to show differences between SCARF results on 2 different Pegasus releases.