Trials

[ianmilligan1]

Issue to hold data on time trials.

[ianmilligan1]

c5.large downloading data: began Dec 18 17:17, finished Dec 19 08:57. 909GB.

[ianmilligan1]

Data handed off to i3.2xlarge, begins derivative processing Dec 19 10:20. 909GB.

• Domains finished at 12:55pm, derivative dataset is 2.3MB.
• Text finished at 4:50pm, derivative dataset is 82GB
• Links finished at 9:50pm, derivative dataset is 33MB

Uploading to S3 buckets is quick from the EBS volume (~ 5 minutes to upload all derivatives).

[ianmilligan1]

Trial #2

c5.large downloading data: began Dec 21 16:28, finished Dec 23 12:49am. 3.5TB.

Data handed off to r3.2xlarge, begins derivative processing Dec 21 12:58am.

• Domains finished at Dec 24 9:59am, derivative dataset is 9.4MB.
• Text failed. (will dump log and open issue on AUT – looks like it is the 538th file)

I then killed this.

Note: this collection failed before on Compute Canada, so we'll see how it shakes out.

I'm happy with the resource utilization on this machine:

[ianmilligan1]

Trial #3

I did this on an Azure machine, 16 core, 55 GB RAM. 206GB. Canadian Federal Political Candidates collection. Previously failed on WALK machine.

• Began 15:26:32
• Domains finished 16:30. Derivative size 2.6MB.
• Text finished 17:32. Derivative size 19GB.
• Links finished 18:36. Derivative size 13MB.

[ianmilligan1]

Trial #4

I did this on an Azure machine, 16 core, 55 GB RAM. 293GB. Toronto Mayoral Election 2015 collection. Previously failed on WALK machine.

• Began 19:31:40
• Failed due to java heap space at 19:51:19

has a 7GB WARC. Going to try increasing partition numbers.