search

google / cloud-forensics-utils

Python library to carry out DFIR analysis on the Cloud
Apache License 2.0
453 stars 89 forks source link

[Do not merge] E2E additions and cleanup, part 1 - AWS #353

Closed ramo-j closed 3 years ago

ramo-j commented 3 years ago

Per #349, #350:

Confirmed the tests work locally:

$ python3 tests/providers/aws/e2e_cli.py
[2021-07-23 07:21:41,250] [libcloudforensics.providers.aws.forensics] INFO     No AMI provided, fetching one for Ubuntu 18.04
[2021-07-23 07:21:41,927] [libcloudforensics.providers.aws.forensics] INFO     Starting analysis VM new-vm-for-analysis
[2021-07-23 07:21:42,054] [libcloudforensics.providers.aws.forensics] INFO     VM started.
[2021-07-23 07:21:42,054] [libcloudforensics.providers.aws.forensics] INFO     VM ready.
[2021-07-23 07:21:42,736] [libcloudforensics.providers.aws.forensics] INFO     Volume copy of vol-017a715a3f77a0276 started...
[2021-07-23 07:22:43,266] [libcloudforensics.providers.aws.forensics] INFO     Created snapshot: snap-0461aef1f7384fd27
[2021-07-23 07:23:01,560] [libcloudforensics.providers.aws.forensics] INFO     Volume vol-017a715a3f77a0276 successfully copied to vol-033ea11837d661657
[2021-07-23 07:23:01,560] [libcloudforensics.providers.aws.forensics] INFO     Cleaning up...
[2021-07-23 07:23:01,810] [libcloudforensics.providers.aws.forensics] INFO     Done
....[2021-07-23 07:23:03,217] [libcloudforensics.providers.aws.forensics] INFO     Volume copy of vol-017a715a3f77a0276 started...
[2021-07-23 07:23:33,661] [libcloudforensics.providers.aws.forensics] INFO     Created snapshot: snap-0b43140abc21006e9
[2021-07-23 07:23:52,021] [libcloudforensics.providers.aws.forensics] INFO     Volume vol-017a715a3f77a0276 successfully copied to vol-0963cfe784d3f523f
[2021-07-23 07:23:52,022] [libcloudforensics.providers.aws.forensics] INFO     Cleaning up...
[2021-07-23 07:23:52,281] [libcloudforensics.providers.aws.forensics] INFO     Done
[2021-07-23 07:23:52,287] [libcloudforensics.providers.aws.forensics] INFO     No AMI provided, fetching one for Ubuntu 18.04
[2021-07-23 07:23:53,626] [libcloudforensics.providers.aws.forensics] INFO     Starting analysis VM new-vm-for-analysis
[2021-07-23 07:23:53,720] [libcloudforensics.providers.aws.forensics] INFO     VM started.
[2021-07-23 07:23:53,720] [libcloudforensics.providers.aws.forensics] INFO     Attaching volume vol-0963cfe784d3f523f to device /dev/sdp
[2021-07-23 07:23:54,266] [libcloudforensics.providers.aws.forensics] INFO     VM ready.
.[2021-07-23 07:23:54,785] [libcloudforensics.providers.aws.forensics] INFO     Volume copy of vol-017a715a3f77a0276 started...
[2021-07-23 07:24:25,213] [libcloudforensics.providers.aws.forensics] INFO     Created snapshot: snap-0c27c0f1de5dde291
[2021-07-23 07:24:43,656] [libcloudforensics.providers.aws.forensics] INFO     Volume vol-017a715a3f77a0276 successfully copied to vol-0f5d62324ec39051b
[2021-07-23 07:24:43,656] [libcloudforensics.providers.aws.forensics] INFO     Cleaning up...
[2021-07-23 07:24:43,879] [libcloudforensics.providers.aws.forensics] INFO     Done
.[2021-07-23 07:24:44,296] [libcloudforensics.providers.aws.forensics] INFO     Volume copy of vol-017a715a3f77a0276 started...
[2021-07-23 07:25:14,730] [libcloudforensics.providers.aws.forensics] INFO     Created snapshot: snap-0825ec7673a85dac0
[2021-07-23 07:25:33,104] [libcloudforensics.providers.aws.forensics] INFO     Volume vol-017a715a3f77a0276 successfully copied to vol-0b5bc1d9ed1fb985e
[2021-07-23 07:25:33,105] [libcloudforensics.providers.aws.forensics] INFO     Cleaning up...
[2021-07-23 07:25:33,429] [libcloudforensics.providers.aws.forensics] INFO     Done
.[2021-07-23 07:26:03,977] [__main__            ] INFO     Deleting volume: vol-033ea11837d661657.
[2021-07-23 07:26:19,307] [__main__            ] INFO     Volume vol-033ea11837d661657 successfully deleted.
[2021-07-23 07:26:19,308] [__main__            ] INFO     Deleting volume: vol-0963cfe784d3f523f.
[2021-07-23 07:26:34,646] [__main__            ] INFO     Volume vol-0963cfe784d3f523f successfully deleted.
[2021-07-23 07:26:34,646] [__main__            ] INFO     Deleting volume: vol-0f5d62324ec39051b.
[2021-07-23 07:26:49,987] [__main__            ] INFO     Volume vol-0f5d62324ec39051b successfully deleted.
[2021-07-23 07:26:49,987] [__main__            ] INFO     Deleting volume: vol-0b5bc1d9ed1fb985e.
[2021-07-23 07:27:05,339] [__main__            ] INFO     Volume vol-0b5bc1d9ed1fb985e successfully deleted.
[2021-07-23 07:27:06,756] [libcloudforensics.providers.aws.internal.s3] INFO     Bucket successfully created
[2021-07-23 07:27:06,916] [libcloudforensics.providers.aws.internal.s3] INFO     Removing e2e_cli.py from ramoj-test-bucket-e2e
[2021-07-23 07:27:07,010] [libcloudforensics.providers.aws.internal.s3] INFO     Deleting bucket ramoj-test-bucket-e2e
.[2021-07-23 07:27:07,500] [libcloudforensics.providers.aws.internal.s3] INFO     Deleting bucket ramoj-test-bucket-e2e
[2021-07-23 07:27:07,602] [libcloudforensics.providers.aws.internal.iam] INFO     Creating IAM Instance Profile ebsCopy
[2021-07-23 07:27:08,733] [libcloudforensics.providers.aws.internal.iam] INFO     Creating IAM policy ebsCopy-policy
[2021-07-23 07:27:08,982] [libcloudforensics.providers.aws.internal.iam] INFO     Creating IAM Role ebsCopy-role
[2021-07-23 07:27:09,238] [libcloudforensics.providers.aws.internal.iam] INFO     Attaching policy arn:aws:iam::752954844773:policy/ebsCopy-policy to role ebsCopy-role
[2021-07-23 07:27:09,486] [libcloudforensics.providers.aws.internal.iam] INFO     Attaching role ebsCopy-role to instance profile ebsCopy
[2021-07-23 07:27:09,736] [libcloudforensics.providers.aws.forensics] INFO     Finding AMI
[2021-07-23 07:27:30,609] [libcloudforensics.providers.aws.forensics] INFO     Starting copy instance
[2021-07-23 07:27:32,052] [libcloudforensics.providers.aws.forensics] INFO     Checking for output files with exponential backoff
[2021-07-23 07:27:32,052] [libcloudforensics.providers.aws.forensics] INFO     Waiting 10 seconds
[2021-07-23 07:27:42,342] [libcloudforensics.providers.aws.forensics] INFO     Waiting 20 seconds
[2021-07-23 07:28:02,537] [libcloudforensics.providers.aws.forensics] INFO     Waiting 40 seconds
[2021-07-23 07:28:42,752] [libcloudforensics.providers.aws.forensics] INFO     Waiting 80 seconds
[2021-07-23 07:30:03,007] [libcloudforensics.providers.aws.forensics] INFO     Waiting 160 seconds
[2021-07-23 07:32:43,199] [libcloudforensics.providers.aws.internal.iam] INFO     Detaching role ebsCopy-role from instance profile ebsCopy
[2021-07-23 07:32:44,263] [libcloudforensics.providers.aws.internal.iam] INFO     Detaching policy arn:aws:iam::752954844773:policy/ebsCopy-policy from role ebsCopy-role
[2021-07-23 07:32:44,516] [libcloudforensics.providers.aws.internal.iam] INFO     Deleting instance profile ebsCopy
[2021-07-23 07:32:44,792] [libcloudforensics.providers.aws.internal.iam] INFO     Deleting IAM role ebsCopy-role
[2021-07-23 07:32:45,070] [libcloudforensics.providers.aws.internal.iam] INFO     Deleting IAM policy arn:aws:iam::752954844773:policy/ebsCopy-policy
[2021-07-23 07:32:45,363] [libcloudforensics.providers.aws.forensics] INFO     Image and hash copied to s3://ramoj-test-bucket/snapshots/snap-0aee13cd5485ba73f/
[2021-07-23 07:32:45,512] [libcloudforensics.providers.aws.internal.s3] INFO     Removing snapshots/snap-0aee13cd5485ba73f/image.bin from ramoj-test-bucket
[2021-07-23 07:32:45,584] [libcloudforensics.providers.aws.internal.s3] INFO     Removing snapshots/snap-0aee13cd5485ba73f/log.txt from ramoj-test-bucket
[2021-07-23 07:32:45,689] [libcloudforensics.providers.aws.internal.s3] INFO     Removing snapshots/snap-0aee13cd5485ba73f/hlog.txt from ramoj-test-bucket
[2021-07-23 07:32:45,722] [libcloudforensics.providers.aws.internal.s3] INFO     Removing snapshots/snap-0aee13cd5485ba73f/mlog.txt from ramoj-test-bucket
.
----------------------------------------------------------------------
Ran 9 tests in 664.519s

OK

Closes #349

ramo-j commented 3 years ago

Blocked by https://github.com/google/cloud-forensics-utils/issues/352

giovannt0 commented 3 years ago

Mmmh. Why did you delete all the work I just did in https://github.com/google/cloud-forensics-utils/pull/314? :D We want to test the CLI. Since the CLI calls the underlying forensics package, testing this covers more ground than the current e2e.py file. So, for your S3 tests, what you should add is a test in e2e_cli.py that goes through the CLI as the other tests in that file. Once that works, we will deprecate e2e.py and only add new e2e tests that go through the CLI. Let me know if this is unclear, happy to chat further.

ramo-j commented 3 years ago

Closing, will discuss OOB.