search

archivematica / Issues

Issues repository for the Archivematica project
GNU Affero General Public License v3.0
16 stars 1 forks source link

Problem: automated acceptance tests have not been run against v1.7.2-rc.4 #139

Closed jrwdunham closed 5 years ago

jrwdunham commented 6 years ago

Please describe the problem you'd like to be solved.

Archivematica's automated user acceptance test suite (AMAUAT) should be run against the latest Archivematica v.1.7.2 release candidate prior to cutting the release. This will help us to discover regressions. Cf. https://github.com/artefactual/archivematica/issues/942.

Describe the solution you'd like to see implemented.

I would like to see the following Archivematica deploy tested with AMAUAT and the results documented in this issue:

The following tests were run against an Archivematica instance deployed locally with the above two commits using the am.git Docker Compose method. The AMAUAT tests were run from the host machine. Here are the details of the testing environment:

jrwdunham commented 6 years ago

Tests to Run

Passing

Passing after fix

Failing

Skipped

The following features under core/ were skipped for the reasons supplied in parentheses after the feature name.

jrwdunham commented 6 years ago

metadata-only-aip-reingest.feature

STATUS: PASSING

Command

$ behave \
      --tags=mo-aip-reingest \
      --no-skipped \
      -v \
      -D am_username=test \
      -D am_password=test \
      -D am_url=http://127.0.0.1:62080/ \
      -D am_version=1.7 \
      -D am_api_key=test \
      -D ss_username=test \
      -D ss_password=test \
      -D ss_url=http://127.0.0.1:62081/ \
      -D ss_api_key=test \
      -D home=archivematica \
      -D driver_name=Firefox

Result

1 feature passed, 0 failed, 19 skipped
1 scenario passed, 0 failed, 51 skipped
36 steps passed, 0 failed, 522 skipped, 0 undefined
Took 4m35.652s
jrwdunham commented 6 years ago

ingest-policy-check.feature

STATUS: PASSING

Command

 $ behave \
      --tags=ipc \
      -D am_url=http://127.0.0.1:62080/ \
      -D ss_url=http://127.0.0.1:62081/ \
      -D am_username=test \
      -D am_password=test \
      -D ss_username=test \
      -D ss_password=test \
      --no-skipped \
      -D driver_name=Firefox \
      -D transfer_source_path=archivematica/archivematica-sampledata/TestTransfers/acceptance-tests \
      -D am_version=1.7

Result

On the first run, the @preservation @nonmanual feature failed on the all-conform-policy because of a circumstantial failure of the tests, viz. RecursionError: maximum recursion depth exceeded while calling a Python object was triggered by the recursive am_browser_jobs_tasks_ability.py::get_job_uuid. This is reflected in the failed scenario below:

0 features passed, 1 failed, 19 skipped
7 scenarios passed, 1 failed, 44 skipped
108 steps passed, 1 failed, 449 skipped, 0 undefined
Took 22m1.490s

On a second run (i.e., adding --tags=preservation --tags=nonmanual to the above command), the failing scenario passed:

1 feature passed, 0 failed, 19 skipped
2 scenarios passed, 0 failed, 50 skipped
32 steps passed, 0 failed, 526 skipped, 0 undefined
Took 4m55.505s
jrwdunham commented 6 years ago

ingest-mkv-conformance.feature

STATUS: PASSING

Command

$ behave \
      --tags=icc \
      -D am_url=http://127.0.0.1:62080/ \
      -D ss_url=http://127.0.0.1:62081/ \
      -D am_username=test \
      -D am_password=test \
      -D ss_username=test \
      -D ss_password=test \
      --no-skipped \
      -D driver_name=Firefox \
      -D transfer_source_path=archivematica/archivematica-sampledata/TestTransfers/acceptance-tests \
      -D am_version=1.7

Result

1 feature passed, 0 failed, 19 skipped
2 scenarios passed, 0 failed, 50 skipped
19 steps passed, 0 failed, 539 skipped, 0 undefined
Took 4m23.470s
jrwdunham commented 6 years ago

transfer-policy-check.feature

STATUS: PASSING

Command

Note: the AM/SS URLs, which are different from those in the commands above. This is a result of updating my am.git.

$ behave \
      --tags=tpc \
      -D am_url=http://127.0.0.1:61780/ \
      -D ss_url=http://127.0.0.1:61781/ \
      -D am_username=test \
      -D am_password=test \
      -D ss_username=test \
      -D ss_password=test \
      --no-skipped \
      -D driver_name=Firefox \
      -D transfer_source_path=archivematica/archivematica-sampledata/TestTransfers/acceptance-tests \
      -D am_version=1.7

Result

1 feature passed, 0 failed, 19 skipped
2 scenarios passed, 0 failed, 50 skipped
26 steps passed, 0 failed, 532 skipped, 0 undefined
Took 4m27.059s
jrwdunham commented 6 years ago

transfer-mkv-conformance.feature

STATUS: PASSING

Command

$ behave \
      --tags=picc \
      -D am_url=http://127.0.0.1:61780/ \
      -D ss_url=http://127.0.0.1:61781/ \
      -D am_username=test \
      -D am_password=test \
      -D ss_username=test \
      -D ss_password=test \
      --no-skipped \
      -D driver_name=Firefox \
      -D transfer_source_path=archivematica/archivematica-sampledata/TestTransfers/acceptance-tests \
      -D am_version=1.7

Result

1 feature passed, 0 failed, 19 skipped
2 scenarios passed, 0 failed, 50 skipped
14 steps passed, 0 failed, 544 skipped, 0 undefined
Took 3m43.780s
jrwdunham commented 6 years ago

uuids-for-directories.feature

STATUS: PASSING

Command

$ behave \
      --tags=uuids-dirs \
      -D am_url=http://127.0.0.1:61780/ \
      -D ss_url=http://127.0.0.1:61781/ \
      -D am_username=test \
      -D am_password=test \
      -D ss_username=test \
      -D ss_password=test \
      --no-skipped \
      -D driver_name=Firefox \
      -D transfer_source_path=archivematica/archivematica-sampledata/TestTransfers/acceptance-tests \
      -D am_version=1.7 \
      -D docker_compose_path=/path/to/dir/containing/docker/compose/file/ \
      -D home=archivematica

Warning: not idempotent

The uuids-for-directories.feature is not idempotent. That is, you cannot run it more than once and expect it to pass. This is because it involves starting a Zipped Bag type transfer and such transfers cannot be given unique names using Archivematica: their names are based on the basename of the .zip file used as the transfer source. Therefore, in order to run this feature multiple times, you have to close all transfers and ingests (or at least the ones named "BagTransfer"). You can do this with the following two commands:

Close all transfers:

$ ./close_all_transfers.sh \
      -D am_url=http://127.0.0.1:61780/ \
      -D ss_url=http://127.0.0.1:61781/ \
      -D am_username=test \
      -D am_password=test \
      -D ss_username=test \
      -D ss_password=test \
      -D driver_name=Firefox \
      -D am_version=1.7

Close all ingests:

$ ./close_all_ingests.sh \
      -D am_url=http://127.0.0.1:61780/ \
      -D ss_url=http://127.0.0.1:61781/ \
      -D am_username=test \
      -D am_password=test \
      -D ss_username=test \
      -D ss_password=test \
      -D driver_name=Firefox \
      -D am_version=1.7

Result

1 feature passed, 0 failed, 19 skipped
2 scenarios passed, 0 failed, 50 skipped
12 steps passed, 0 failed, 546 skipped, 0 undefined
Took 3m25.053s
jrwdunham commented 6 years ago

premis-events.feature

STATUS: PASSING

Command

$ behave \
      --tags=premis-events \
      --no-skipped \
      -v \
      -D am_username=test \
      -D am_password=test \
      -D am_url=http://127.0.0.1:61780/ \
      -D am_version=1.7 \
      -D am_api_key=test \
      -D ss_username=test \
      -D ss_password=test \
      -D ss_url=http://127.0.0.1:61781/ \
      -D ss_api_key=test \
      -D home=archivematica \
      -D driver_name=Firefox

Result

1 feature passed, 0 failed, 19 skipped
5 scenarios passed, 0 failed, 47 skipped
86 steps passed, 0 failed, 472 skipped, 0 undefined
Took 9m58.158s
jrwdunham commented 6 years ago

pid-binding.feature

STATUS: FAILING

Command

$ behave \
    --tags=pid-binding \
    --no-skipped \
    -D am_username=test \
    -D am_password=test \
    -D am_url=http://127.0.0.1:61780/ \
    -D am_api_key=test \
    -D ss_username=test  \
    -D ss_password=test \
    -D ss_url=http://127.0.0.1:61781/ \
    -D ss_api_key=test \
    -D home=archivematica \
    -D driver_name=Firefox \
    -D am_version=1.7 \
    -D docker_compose_path=/path/to/dir/containing/docker/compose/file/ \
    -D pid_web_service_endpoint=<SOME_URL> \
    -D pid_web_service_key=<SOME_SECRET> \
    -D handle_resolver_url=<SOME_RESOLVER_URL> \
    -D base_resolve_url=<SOME_RESOLVE_URL> \
    -D pid_xml_namespace=<SOME_NAMESPACE>

Result

0 features passed, 1 failed, 19 skipped
0 scenarios passed, 1 failed, 51 skipped
7 steps passed, 1 failed, 550 skipped, 0 undefined
Took 1m59.168s

The failure is presumably caused by the supplied handle credentials being invalid:

Then the AIP METS file documents PIDs, PURLs, and UUIDs for all files, directories and the package itself                                                                                                                
        # features/steps/pid_binding_steps.py:97 12.018s
    Assertion Failed: Unable to find an identifier of type hdl for entity hierarchy_with_empty_dir_1536189083-f45125dc-69a7-42ac-8246-7e88ed519a91
jrwdunham commented 6 years ago

Change in how the tests were run

The following AMAUAT features were run against an Archivematica instance deployed on OVH (Ubuntu Xenial) using deploy-pub (Vagrant/Ansible). The steps used to do this are given below.

First, install the vagrant openstack provider::

$ vagrant plugin install vagrant-openstack-provider

Then checkout the dev/vagrant-openstack branch of deploy-pub::

$ pwd
/.../deploy-pub/playbooks/archivematica-xenial
$ git st
On branch dev/vagrant-openstack

Make some mods to the Vagrantfile. (Note: using the "xenial_template1" makes the provisioning much faster, but it failed for me so that's why the change to "Ubuntu 16.04".)::

$ git diff
diff --git a/playbooks/archivematica-xenial/Vagrantfile b/playbooks/archivematica-xenial/Vagrantfile
index c8b8329..bdc7bfe 100644
--- a/playbooks/archivematica-xenial/Vagrantfile
+++ b/playbooks/archivematica-xenial/Vagrantfile
@@ -22,8 +22,10 @@ Vagrant.configure('2') do |config|
    os.password                         = ENV['OS_PASSWORD']
    os.region                           = ENV['OS_REGION_NAME']
    os.flavor                           = "b2-7"
-    os.image                            = "xenial_template1"
+    #os.image                            = "xenial_template1"
+    os.image                            = "Ubuntu 16.04"
    ov.nfs.functional = false
+    ov.vm.allowed_synced_folder_types = :rsync
end

Download the Ansible roles::

$ ansible-galaxy install -f -p roles/ -r requirements.yml

Bring up the OVH VM and provision it::

$ vagrant up

Get the SSH details::

$ vagrant ssh-config
Host default
  HostName XXX.XX.XXX.XXX
  User ubuntu
  IdentityFile /.../deploy-pub/playbooks/archivematica-xenial/.vagrant/machines/default/openstack/vagrant-generated-eh6rotde

Add the ubuntu user to the archivematica group::

ubuntu@test-xenial:~$ sudo adduser ubuntu archivematica
Adding user `ubuntu' to group `archivematica' ...
Adding user ubuntu to group archivematica
Done.
jrwdunham commented 6 years ago

aip-encryption-mirror.feature

STATUS: PASSING

Command

$ behave \
      --tags=aip-encrypt-mirror \
      --no-skipped \
      -D am_version=1.7 \
      -D driver_name=Firefox \
      -D am_username=admin \
      -D am_password=archivematica \
      -D am_url=http://XXX.XX.XXX.XXX/ \
      -D ss_username=admin \
      -D ss_password=archivematica \
      -D ss_url=http://XXX.XX.XXX.XXX:8000/ \
      -D home=ubuntu \
      -D server_user=ubuntu \
      -D ssh_identity_file=/.../deploy-pub/playbooks/archivematica-xenial/.vagrant/machines/default/openstack/vagrant-generated-eh6rotde

Result

1 feature passed, 0 failed, 19 skipped
1 scenario passed, 0 failed, 51 skipped
25 steps passed, 0 failed, 533 skipped, 0 undefined
Took 3m8.262s

Notes

The first attempt to run this feature failed on the second step of the sole scenario And there is a standard GPG-encrypted Replicator location in the storage service with an error in the GUI involving a missing value for a required field. @scollazo saw the same issue in a separate run of this feature.

jrwdunham commented 6 years ago

aip-encryption.feature

Attempt 1—against SS v0.12.0-rc.2

STATUS: FAILING

Command

$ behave \
      --tags=aip-encrypt \
      --no-skipped \
      -D am_version=1.7 \
      -D driver_name=Firefox \
      -D am_username=admin \
      -D am_password=archivematica \
      -D am_url=http://XXX.XX.XXX.XXX/ \
      -D ss_username=admin \
      -D ss_password=archivematica \
      -D ss_url=http://XXX.XX.XXX.XXX:8000/ \
      -D home=ubuntu \
      -D server_user=ubuntu \
      -D ssh_identity_file=/.../deploy-pub/playbooks/archivematica-xenial/.vagrant/machines/default/openstack/vagrant-generated-eh6rotde

Result

Attempting to store a (compressed) AIP in a GPG space will fail at Store AIP: Store the AIP.

Attempt 2—against SS 07ae87501b93f4c917f1c0a49dc76ba5c369039d (PR 409)

STATUS: PASSING

Command

Same as above.

Result

1 feature passed, 0 failed, 19 skipped
7 scenarios passed, 0 failed, 45 skipped
59 steps passed, 0 failed, 499 skipped, 0 undefined
Took 22m5.523s
jrwdunham commented 5 years ago

capture-output-option.feature

STATUS: PASSING

Command

With output capturing enabled in AM:

$ behave -v --tags=capture-stdout \
      --tags=deploy.output_capture_state.on \
      --no-skipped \
      -D am_username=admin \
      -D am_password=archivematica \
      -D am_url=http://XXX.XX.XXX.XXX/ \
      -D am_version=1.7 \
      -D ss_username=admin \
      -D ss_password=archivematica \
      -D ss_url=http://XXX.XX.XXX.XXX:8000/ \
      -D home=ubuntu \
      -D driver_name=Firefox

With output capturing disabled in AM:

$ behave -v --tags=capture-stdout \
      --tags=deploy.output_capture_state.off \
      --no-skipped \
      -D am_username=admin \
      -D am_password=archivematica \
      -D am_url=http://XXX.XX.XXX.XXX/ \
      -D am_version=1.7 \
      -D ss_username=admin \
      -D ss_password=archivematica \
      -D ss_url=http://XXX.XX.XXX.XXX:8000/ \
      -D home=ubuntu \
      -D driver_name=Firefox

Note: to disable output capturing in this type of OVH Vagrant/Ansible Ubuntu 16 deploy, add ARCHIVEMATICA_MCPCLIENT_MCPCLIENT_CAPTURE_CLIENT_SCRIPT_OUTPUT=false to /etc/default/archivematica-mcp-client:

$ cat /etc/default/archivematica-mcp-client
...
ARCHIVEMATICA_MCPCLIENT_MCPCLIENT_CAPTURE_CLIENT_SCRIPT_OUTPUT=false
...

and restart the MCP client service:

$ sudo service archivematica-mcp-client stop
$ sudo service archivematica-mcp-client start

Result

With output capturing enabled in AM:

1 feature passed, 0 failed, 19 skipped
1 scenario passed, 0 failed, 51 skipped
4 steps passed, 0 failed, 554 skipped, 0 undefined
Took 1m1.208s

With output capturing disabled in AM:

1 feature passed, 0 failed, 19 skipped
1 scenario passed, 0 failed, 51 skipped
4 steps passed, 0 failed, 554 skipped, 0 undefined
Took 1m1.962s
jrwdunham commented 5 years ago

indexless.feature

STATUS: PASSING

Command

$ behave \
      --tags=deploy.method.ansible,same-aip,gui \
      --no-skipped \
      -D am_version=1.7 \
      -D driver_name=Firefox \
      -D am_username=admin \
      -D am_password=archivematica \
      -D am_url=http://XXX.XX.XXX.XXX/ \
      -D ss_username=admin \
      -D ss_password=archivematica \
      -D ss_url=http://XXX.XX.XXX.XXX:8000/ \
      -D home=ubuntu

Note 1: Indexless Mode

To put this OVH Vagrant/Ansible Ubuntu 16 deploy in indexless mode, set the following env vars as described and restart the relevant services:

$ cat /etc/default/archivematica-dashboard
ARCHIVEMATICA_DASHBOARD_DASHBOARD_SEARCH_ENABLED=false
$ cat /etc/default/archivematica-mcp-server
ARCHIVEMATICA_MCPSERVER_MCPSERVER_SEARCH_ENABLED=false
$ cat /etc/default/archivematica-mcp-client
ARCHIVEMATICA_MCPCLIENT_MCPCLIENT_SEARCH_ENABLED=false
$ sudo service archivematica-dashboard stop
$ sudo service archivematica-dashboard start
$ sudo service archivematica-mcp-server stop
$ sudo service archivematica-mcp-server start
$ sudo service archivematica-mcp-client stop
$ sudo service archivematica-mcp-client start

Note 2: Disable the "ES not installed" step

Since we are simply disabling indexing on a deploy that already has ES installed, it does not make sense for the feature to assert that ES is not installed. Therefore, comment out the step that does that:

$ git diff features/
diff --git a/features/core/indexless.feature b/features/core/indexless.feature
index 09f729e..365c53e 100644
--- a/features/core/indexless.feature
+++ b/features/core/indexless.feature
@@ -66,7 +66,7 @@ Feature: Archivematica is deployed without Indexing feature
     Given Archivematica deployment method <method> which provides an option to exclude Elasticsearch
     When Archivematica is deployed without Elasticsearch using method <method>
     Then the installation has Indexing disabled
-    And Elasticsearch is not installed
+    #And Elasticsearch is not installed

Result

1 feature passed, 0 failed, 19 skipped
3 scenarios passed, 0 failed, 49 skipped
33 steps passed, 0 failed, 522 skipped, 0 undefined
Took 4m27.164s
jrwdunham commented 5 years ago

manual-normalization.feature

Command

$ behave --tags=man-norm \
      --no-skipped \
      -D am_version=1.7 \
      -D driver_name=Firefox \
      -D am_username=admin \
      -D am_password=archivematica \
      -D am_url=http://XXX.XX.XXX.XXX/ \
      -D ss_username=admin \
      -D ss_password=archivematica \
      -D ss_url=http://XXX.XX.XXX.XXX:8000/ \
      -D home=ubuntu

Attempt 1—against AM v1.7.2-rc.4

STATUS: FAILING

Result

0 features passed, 1 failed, 19 skipped
0 scenarios passed, 1 failed, 51 skipped
5 steps passed, 1 failed, 549 skipped, 0 undefined
Took 19m52.671s

AM (normalize.py) is failing to recognize that %SIPDirectory%objects/image.bmp has a manually normalized preservation derivative in %SIPDirectory%objects/manualNormalization/preservation/image.tif. See https://github.com/archivematica/Issues/issues/150.

Attempt 2—against AM at 26a261c4e4c43cdf091ddf1a027657b9ac8fffaa

STATUS: PASSING

Result

This is with cherry-picking the fix from https://github.com/artefactual/archivematica/pull/973 into stable/1.7.x.

1 feature passed, 0 failed, 19 skipped
1 scenario passed, 0 failed, 51 skipped
11 steps passed, 0 failed, 547 skipped, 0 undefined
Took 5m38.030s
Note

This requires a fix to the manual-normalization.feature so that the "Document empty directories" decision is provided. This fix is provided in https://github.com/artefactual-labs/archivematica-acceptance-tests/pull/128.

jrwdunham commented 5 years ago

mediaconch-challenging-mkv.feature

STATUS: PASSING

Command

$ behave \
      --tags=mediaconch-challenging-mkv \
      --no-skipped \
      -D am_version=1.7 \
      -D driver_name=Firefox \
      -D am_username=admin \
      -D am_password=archivematica \
      -D am_url=http://XXX.XX.XXX.XXX/ \
      -D ss_username=admin \
      -D ss_password=archivematica \
      -D ss_url=http://XXX.XX.XXX.XXX:8000/ \
      -D home=ubuntu

Result

1 feature passed, 0 failed, 19 skipped
1 scenario passed, 0 failed, 51 skipped
10 steps passed, 0 failed, 548 skipped, 0 undefined
Took 6m37.841s

Notes

On the first run, this failed at Then all PREMIS implementation-check-type validation events have eventOutcome = pass. This seems to have been an issue with the tests themselves and not with AM.

jrwdunham commented 5 years ago

Results