Testing integration

[mturilli]

Testing real use case and deployment.

TODO:

• Bulk pull from RP
• Data staging #11
• target an XSEDE resource

[dgasmith]

The current jobs on the RP two instance are representative of the functionality that we will need, but production runs will be much more time per job. If you can provide some details on the number of core hours (1e4-1e6 hours), number of tasks (1e4-1e6 tasks), and run time of task (0.1-100 core hours average) that would be optimal, we can provide a production level use for you.

We truly do have an extremely wide variety of production ready use cases here.

[andre-merzky]

Hi Daniel - we also can handle a variety of workloads. Those are our constraints:

• pilot runtime should not exceed 12 hrs, must not exceed 24 hrs
• task number should not exceed 16k per pilot, must not exceed 64k per pilot
• shorter tasks should run on smaller pilots: 1s/task -> 128 cores per pilot; 30min/task -> 64k cores per pilot
• smaller pilots have a less startup overhead - but up to about 1k cores its almost constant.
• there is no hard limit on pilot size

HtH, Andre.

[andre-merzky]

This is now implemented here. The code pulls bulkds of 500 tasks, executes them, and pushes the result-bulk back to QCArchive. Semantically this is the real deal - next it targeting some remote compute resource (which then requires proper data staging), and to start optimizing execution performance.

[mturilli]

@dgasmith and @andre-merzky to try to run the integration locally and remotely.

[andre-merzky]

Update: I confirmed that a pilot is correctly instantiated on a remote resource (comet), but did not test full integration.

[Lnaden]

Determined that if it installs, thats good enough for now. We can open a new ticket if new issues arise