Large numbers of input files lead to unacceptable setup time

payu-org / payu

A workflow management tool for numerical models on the NCI computing systems

Apache License 2.0

18 stars 26 forks source link

Closed aidanheerdegen closed 4 years ago

aidanheerdegen commented 4 years ago

User of MITgcm has 6TB of input files and the manifest calculation was over an hour before it was killed.

This is unacceptable.

Solutions include being able to specify only fast hashing, or turning off hashing altogether