Reduce duplicate files

[mmsrubar]

According to the question in #771 I ran a scan to find duplicate files in SSG. You can take a look at the result:

https://msrubar.fedorapeople.org/ssg-duplicates.out

I think it would be worth the time to take a closer look at it. What do you think?

[shawndwells]

@mmsrubar do you have the script used to generate this? would be interesting to include in the build system

[mmsrubar]

I think I used some utility like fslint or fdupes but I don't really remember now ...

[ybznek]

I wrote my own quick&dirty script

#!/bin/bash

SSG_PATH=/ssg-new/scap-security-guide

tmp=$(mktemp)

find "$SSG_PATH" -type f -not -path './.git/*' -exec md5sum {} \; 2>/dev/null | grep -v git > "$tmp"
cut -f 1 -d' ' "$tmp" | sort | uniq -c | grep -v "^\\s*1\\s" | while read count hash;
do
    echo "${count}x: ${hash}"
    grep "$hash" "$tmp" | awk '{ print $2 }' | sed "s;^;\t;g"
done

rm "$tmp"

2x: 0bbdbd2bb1e1c62a21aa4d94bd6f50a0
    /ssg-new/scap-security-guide/JBoss/EAP/5/utils/sync-alt-titles.py
    /ssg-new/scap-security-guide/JBoss/Fuse/6/utils/sync-alt-titles.py
4x: 0fc3b3ca4e51793d3df22e0f8b061d57
    /ssg-new/scap-security-guide/Chromium/transforms/splitchecks.py
    /ssg-new/scap-security-guide/Firefox/transforms/splitchecks.py
    /ssg-new/scap-security-guide/JRE/transforms/splitchecks.py
    /ssg-new/scap-security-guide/RHEVM3/transforms/splitchecks.py
2x: 2103aef35c5fdc08f9ffd309e95d38f0
    /ssg-new/scap-security-guide/JBoss/EAP/5/utils/verify-cce.py
    /ssg-new/scap-security-guide/JBoss/Fuse/6/utils/verify-cce.py
3x: 2b256a51d967aee2a2272bdb6f42d462
    /ssg-new/scap-security-guide/JBoss/EAP/5/transforms/xccdf2csv-stig.py
    /ssg-new/scap-security-guide/JBoss/Fuse/6/transforms/xccdf2csv-stig.py
    /ssg-new/scap-security-guide/OpenStack/RHEL-OSP/7/transforms/xccdf2csv-stig.py
7x: 38043533320658265fb8a3aa6d83b31c
    /ssg-new/scap-security-guide/RHEL/5/utils/sync-alt-titles.py
    /ssg-new/scap-security-guide/RHEL/6/utils/sync-alt-titles.py
    /ssg-new/scap-security-guide/RHEL/7/utils/sync-alt-titles.py
    /ssg-new/scap-security-guide/Webmin/utils/sync-alt-titles.py
    /ssg-new/scap-security-guide/OpenStack/RHEL-OSP/7/utils/sync-alt-titles.py
    /ssg-new/scap-security-guide/SUSE/11/utils/sync-alt-titles.py
    /ssg-new/scap-security-guide/SUSE/12/utils/sync-alt-titles.py
3x: 3d2c73ee9b7a6f66f87058c579bcb466
    /ssg-new/scap-security-guide/JBoss/EAP/5/transforms/splitchecks.py
    /ssg-new/scap-security-guide/JBoss/Fuse/6/transforms/splitchecks.py
    /ssg-new/scap-security-guide/OpenStack/RHEL-OSP/7/transforms/splitchecks.py
6x: 454564b6572f60cdc77b78ef24d68409
    /ssg-new/scap-security-guide/RHEL/5/transforms/cce_extract.py
    /ssg-new/scap-security-guide/RHEL/6/transforms/cce_extract.py
    /ssg-new/scap-security-guide/RHEL/7/transforms/cce_extract.py
    /ssg-new/scap-security-guide/Webmin/transforms/cce_extract.py
    /ssg-new/scap-security-guide/SUSE/11/transforms/cce_extract.py
    /ssg-new/scap-security-guide/SUSE/12/transforms/cce_extract.py
4x: 49cbf09d31eb25ce42a1d5e8ef42c4a6
    /ssg-new/scap-security-guide/Chromium/utils/sync-alt-titles.py
    /ssg-new/scap-security-guide/Firefox/utils/sync-alt-titles.py
    /ssg-new/scap-security-guide/JRE/utils/sync-alt-titles.py
    /ssg-new/scap-security-guide/RHEVM3/utils/sync-alt-titles.py
6x: 5a76a3f287f8f350b1222365d83a224f
    /ssg-new/scap-security-guide/RHEL/5/transforms/splitchecks.py
    /ssg-new/scap-security-guide/RHEL/6/transforms/splitchecks.py
    /ssg-new/scap-security-guide/RHEL/7/transforms/splitchecks.py
    /ssg-new/scap-security-guide/Webmin/transforms/splitchecks.py
    /ssg-new/scap-security-guide/SUSE/11/transforms/splitchecks.py
    /ssg-new/scap-security-guide/SUSE/12/transforms/splitchecks.py
4x: 5c6074c2ced683c5e1ea6c89ba499376
    /ssg-new/scap-security-guide/Chromium/transforms/xccdf2csv-stig.py
    /ssg-new/scap-security-guide/Firefox/transforms/xccdf2csv-stig.py
    /ssg-new/scap-security-guide/JRE/transforms/xccdf2csv-stig.py
    /ssg-new/scap-security-guide/RHEVM3/transforms/xccdf2csv-stig.py
6x: 75e220d3ceb5f081a1e924ce29338b1d
    /ssg-new/scap-security-guide/RHEL/5/transforms/xccdf2csv-stig.py
    /ssg-new/scap-security-guide/RHEL/6/transforms/xccdf2csv-stig.py
    /ssg-new/scap-security-guide/RHEL/7/transforms/xccdf2csv-stig.py
    /ssg-new/scap-security-guide/Webmin/transforms/xccdf2csv-stig.py
    /ssg-new/scap-security-guide/SUSE/11/transforms/xccdf2csv-stig.py
    /ssg-new/scap-security-guide/SUSE/12/transforms/xccdf2csv-stig.py
4x: 918b3676ef0ce941643704d855d56d82
    /ssg-new/scap-security-guide/Chromium/transforms/cce_extract.py
    /ssg-new/scap-security-guide/Firefox/transforms/cce_extract.py
    /ssg-new/scap-security-guide/JRE/transforms/cce_extract.py
    /ssg-new/scap-security-guide/RHEVM3/transforms/cce_extract.py
3x: b56fb2f3d6e515c311b06586ebd5038a
    /ssg-new/scap-security-guide/JBoss/EAP/5/transforms/cce_extract.py
    /ssg-new/scap-security-guide/JBoss/Fuse/6/transforms/cce_extract.py
    /ssg-new/scap-security-guide/OpenStack/RHEL-OSP/7/transforms/cce_extract.py
7x: c463c1b5541db7d89ffb6e7b0e5f8c16
    /ssg-new/scap-security-guide/RHEL/5/utils/verify-cce.py
    /ssg-new/scap-security-guide/RHEL/6/utils/verify-cce.py
    /ssg-new/scap-security-guide/RHEL/7/utils/verify-cce.py
    /ssg-new/scap-security-guide/Webmin/utils/verify-cce.py
    /ssg-new/scap-security-guide/OpenStack/RHEL-OSP/7/utils/verify-cce.py
    /ssg-new/scap-security-guide/SUSE/11/utils/verify-cce.py
    /ssg-new/scap-security-guide/SUSE/12/utils/verify-cce.py
3x: c7561f1bdda6881b916170060920a6e9
    /ssg-new/scap-security-guide/JBoss/EAP/5/input/oval/testoval.py
    /ssg-new/scap-security-guide/JBoss/Fuse/6/input/oval/testoval.py
    /ssg-new/scap-security-guide/OpenStack/RHEL-OSP/7/input/oval/testoval.py
5x: ca5d890b4192e2b2ac6b3f474bdfe87f
    /ssg-new/scap-security-guide/Chromium/input/oval/testoval.py
    /ssg-new/scap-security-guide/Fedora/input/oval/testoval.py
    /ssg-new/scap-security-guide/Firefox/input/oval/testoval.py
    /ssg-new/scap-security-guide/JRE/input/oval/testoval.py
    /ssg-new/scap-security-guide/WRLinux/input/oval/testoval.py
7x: ce97bca010fdf515af40becd812f844c
    /ssg-new/scap-security-guide/RHEL/5/input/oval/testoval.py
    /ssg-new/scap-security-guide/RHEL/6/input/oval/testoval.py
    /ssg-new/scap-security-guide/RHEL/7/input/oval/testoval.py
    /ssg-new/scap-security-guide/Debian/8/input/oval/testoval.py
    /ssg-new/scap-security-guide/SUSE/11/input/oval/testoval.py
    /ssg-new/scap-security-guide/Ubuntu/14.04/input/oval/testoval.py
    /ssg-new/scap-security-guide/Ubuntu/16.04/input/oval/testoval.py
3x: ecadf096b715cb6e95605fd4e5a9d3d1
    /ssg-new/scap-security-guide/Chromium/utils/verify-cce.py
    /ssg-new/scap-security-guide/Firefox/utils/verify-cce.py
    /ssg-new/scap-security-guide/JRE/utils/verify-cce.py

[redhatrises]

Except for testoval.py which should move to where the all the static OVAL files are kept, I believe that the other python scripts can be removed or moved to shared/utils. They should be tested after moving.

[ybznek]

We have decided to add such test to jenkins build.

[redhatrises]

Closing. The files referenced here are now in shared/utils.