Scrape the Packit configs and check for URLs used in actions

[mfocko]

With the move to the MP+ we're now left with the possible issue of upstream / archive URLs being blocked on the firewall. Goal of this issue is to scrape the used configs and find potentially problematic servers that are not allowed.

TODO:

• [x] Scrape the configs and check for URLs in the actions
• [x] Potentially do the same for the specfiles as they may include sources kept on various servers
• [x] Check the connectivity from MP+
  • [x] You can spin up a pod on packit--stg-sandbox and try fetching the sources
  • [ ] Or use a tool (though the script in a pod might be simpler)

Related to MP+

[majamassarini]

I have done

packit_config_checker.py  download-configs
find . -name "*packit*" -exec cat {} \; | awk '/actions/{f=1} /jobs/{f=0;print} f' | grep http > mpp_uri_test.sh

a bit of hand work and I got this script which I ran in our long running worker on stage with no failures:

curl https://raw.githubusercontent.com/theforeman/foreman-packaging/rpm/develop/packages/plugins/rubygem-foreman_acd/rubygem-foreman_acd.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://raw.githubusercontent.com/theforeman/foreman-packaging/rpm/develop/packages/katello/rubygem-foreman_scc_manager/rubygem-foreman_scc_manager.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://raw.githubusercontent.com/theforeman/foreman-packaging/rpm/develop/packages/plugins/rubygem-foreman_snapshot_management/rubygem-foreman_snapshot_management.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://raw.githubusercontent.com/theforeman/foreman-packaging/rpm/develop/packages/katello/rubygem-katello/rubygem-katello.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; git clone https://github.com/cgwalters/cargo-vendor-filterer.git
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://src.fedoraproject.org/rpms/polkit/raw/rawhide/f/polkit.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://src.fedoraproject.org/rpms/polkit/raw/rawhide/f/polkit.sysusers
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://src.fedoraproject.org/rpms/polkit/raw/rawhide/f/polkit.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://src.fedoraproject.org/rpms/libmks/raw/rawhide/f/libmks.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl -s https://src.fedoraproject.org/rpms/crosswords/raw/main/f/crosswords.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl -s https://src.fedoraproject.org/rpms/libipuz/raw/main/f/libipuz.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://raw.githubusercontent.com/python-poetry/poetry/master/install-poetry.py
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl -O https://src.fedoraproject.org/rpms/conmon/raw/rawhide/f/conmon.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl -O https://src.fedoraproject.org/rpms/conmon/raw/f37/f/conmon.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl -O https://src.fedoraproject.org/rpms/conmon/raw/f36/f/conmon.spe
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://src.fedoraproject.org/rpms/libxcrypt/raw/main/f/libxcrypt.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl -O https://src.fedoraproject.org/rpms/conmon/raw/rawhide/f/conmon.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://src.fedoraproject.org/rpms/cri-o/raw/rawhide/f/cri-o.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; git clone https://src.fedoraproject.org/rpms/dfuzzer --depth=1
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; git clone https://src.fedoraproject.org/rpms/elfutils --depth=1
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; git clone -b packit https://pagure.io/meta/fb303.git
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; git clone -b packit https://pagure.io/meta/folly.git
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl -s https://src.fedoraproject.org/rpms/netconsd/raw/main/f/netconsd.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl -s https://src.fedoraproject.org/rpms/golang-github-facebook-time/raw/main/f/golang-github-facebook-time.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl -s https://src.fedoraproject.org/rpms/golang-github-facebookincubator-go2chef/raw/main/f/golang-github-facebookincubator-go2chef.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl -s https://src.fedoraproject.org/rpms/python-pystemd/raw/main/f/python-pystemd.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl -sSL https://install.python-poetry.org
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://src.fedoraproject.org/rpms/python-fasjson-client/raw/main/f/python-fasjson-client.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl -sSL https://install.python-poetry.org
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://src.fedoraproject.org/rpms/fedora-messaging/raw/main/f/fedora-messaging.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl -sSL https://install.python-poetry.org
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl -o python-flask-oidc.spec https://src.fedoraproject.org/rpms/python-flask-oidc/raw/main/f/python-flask-oidc.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl -sSL https://install.python-poetry.org
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://src.fedoraproject.org/rpms/noggin/raw/rawhide/f/noggin.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://src.fedoraproject.org/rpms/noggin/raw/main/f/noggin.service
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://src.fedoraproject.org/rpms/noggin/raw/main/f/noggin.sysconfig
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://src.fedoraproject.org/rpms/noggin/raw/main/f/sources
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl -sSL https://install.python-poetry.org
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://src.fedoraproject.org/rpms/python-noggin-messages/raw/main/f/python-noggin-messages.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://src.fedoraproject.org/rpms/python-noggin-messages/raw/main/f/0001-Revert-Include-additional-files-in-the-sdist.patch
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://src.fedoraproject.org/rpms/python-noggin-messages/raw/main/f/README.md
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://src.fedoraproject.org/rpms/python-noggin-messages/raw/main/f/sources
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl -sSL https://install.python-poetry.org
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://src.fedoraproject.org/rpms/httpie/raw/rawhide/f/httpie.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://src.fedoraproject.org/rpms/libstoragemgmt/raw/main/f/libstoragemgmt.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://src.fedoraproject.org/rpms/linux-system-roles/raw/rawhide/f/linux-system-roles.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://src.fedoraproject.org/rpms/linux-system-roles/raw/rawhide/f/extrasources.inc
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://src.fedoraproject.org/rpms/linux-system-roles/raw/rawhide/f/ansible-packaging.inc
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://src.fedoraproject.org/rpms/linux-system-roles/raw/rawhide/f/vendoring-prep.inc
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://src.fedoraproject.org/rpms/linux-system-roles/raw/rawhide/f/vendoring-build.inc
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; git clone --branch main --depth 1 https://github.com/dovecot/pigeonhole.git dovecot-pigeonhole
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl -s https://src.fedoraproject.org/rpms/python-drgn/raw/main/f/python-drgn.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://src.fedoraproject.org/rpms/p11-kit/raw/rawhide/f/p11-kit.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://src.fedoraproject.org/rpms/p11-kit/raw/rawhide/f/p11-kit-client.service
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://src.fedoraproject.org/rpms/p11-kit/raw/rawhide/f/trust-extract-compat
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; git clone https://gitlab.com/redhat/centos-stream/rpms/systemd.git --depth=1; rm -rf systemd
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; git clone https://github.com/stratis-storage/ci --depth=1
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; git clone https://src.fedoraproject.org/rpms/systemd --depth=1
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl -s https://src.fedoraproject.org/rpms/python-pystemd/raw/main/f/python-pystemd.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://raw.githubusercontent.com/theforeman/foreman-packaging/rpm/develop/packages/foreman/foreman/foreman.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://raw.githubusercontent.com/theforeman/foreman-packaging/rpm/develop/packages/foreman/foreman/foreman.logrotate
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://raw.githubusercontent.com/theforeman/foreman-packaging/rpm/develop/packages/foreman/foreman/foreman.cron.d
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://raw.githubusercontent.com/theforeman/foreman-packaging/rpm/develop/packages/foreman/foreman/foreman.tmpfiles
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://raw.githubusercontent.com/theforeman/foreman-packaging/rpm/develop/packages/foreman/foreman-installer/foreman-installer.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://raw.githubusercontent.com/theforeman/foreman-packaging/rpm/develop/packages/plugins/rubygem-foreman-tasks/rubygem-foreman-tasks.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://raw.githubusercontent.com/theforeman/foreman-packaging/rpm/develop/packages/plugins/rubygem-foreman-tasks/foreman-tasks.logrotate
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://raw.githubusercontent.com/theforeman/foreman-packaging/rpm/develop/packages/plugins/rubygem-foreman_ansible/rubygem-foreman_ansible.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://raw.githubusercontent.com/theforeman/foreman-packaging/rpm/develop/packages/plugins/rubygem-foreman_bootdisk/rubygem-foreman_bootdisk.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://raw.githubusercontent.com/theforeman/foreman-packaging/rpm/develop/packages/plugins/rubygem-foreman_discovery/rubygem-foreman_discovery.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://raw.githubusercontent.com/theforeman/foreman-packaging/rpm/develop/packages/plugins/rubygem-foreman_leapp/rubygem-foreman_leapp.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://raw.githubusercontent.com/theforeman/foreman-packaging/rpm/develop/packages/foreman/rubygem-foreman_maintain/rubygem-foreman_maintain.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://raw.githubusercontent.com/theforeman/foreman-packaging/rpm/develop/packages/foreman/rubygem-foreman_maintain/foreman_maintain.logrotate
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://raw.githubusercontent.com/theforeman/foreman-packaging/rpm/develop/packages/plugins/rubygem-foreman_openscap/rubygem-foreman_openscap.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://raw.githubusercontent.com/theforeman/foreman-packaging/rpm/develop/packages/plugins/rubygem-foreman_plugin_template/rubygem-foreman_plugin_template.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://raw.githubusercontent.com/theforeman/foreman-packaging/rpm/develop/packages/plugins/rubygem-foreman_puppet/rubygem-foreman_puppet.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://raw.githubusercontent.com/theforeman/foreman-packaging/rpm/develop/packages/plugins/rubygem-foreman_remote_execution/rubygem-foreman_remote_execution.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://raw.githubusercontent.com/theforeman/foreman-packaging/rpm/develop/packages/plugins/rubygem-foreman_salt/rubygem-foreman_salt.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://raw.githubusercontent.com/theforeman/foreman-packaging/rpm/develop/packages/plugins/rubygem-foreman_templates/rubygem-foreman_templates.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://raw.githubusercontent.com/theforeman/foreman-packaging/rpm/develop/packages/plugins/rubygem-foreman_webhooks/rubygem-foreman_webhooks.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://raw.githubusercontent.com/theforeman/foreman-packaging/rpm/develop/packages/foreman/rubygem-hammer_cli/rubygem-hammer_cli.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://raw.githubusercontent.com/theforeman/foreman-packaging/rpm/develop/packages/foreman/rubygem-hammer_cli_foreman/rubygem-hammer_cli_foreman.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://raw.githubusercontent.com/theforeman/foreman-packaging/rpm/develop/packages/plugins/rubygem-hammer_cli_foreman_ansible/rubygem-hammer_cli_foreman_ansible.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://raw.githubusercontent.com/theforeman/foreman-packaging/rpm/develop/packages/plugins/rubygem-smart_proxy_ansible/rubygem-smart_proxy_ansible.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; git clone https://src.fedoraproject.org/rpms/util-linux.git --depth=1
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi

@mfocko do we want to check also the specfile(s) even though it seems not to be problems accessing http uri?

[mfocko]

@majamassarini I'll have a look today, I must've missed the notification /o\

[mfocko]

Thanks a lot, I've went through the specfiles too and found these domains that hold sources and we cannot access them:

• go.dev
• pigeonhole.dovecot.org
• dovecot.org

I'll create a ticket for the IT to enable those.

[majamassarini]

Thanks a lot, I've went through the specfiles too and found these domains that hold sources and we cannot access them:

Oh thank you, I didn't mean you to do this 😅 Thank a lot to you for having done it!