globus / globus-compute

Globus Compute: High Performance Function Serving for Science
https://www.globus.org/compute
Apache License 2.0
146 stars 47 forks source link

Run tests on Bluewaters NCSA #341

Open yadudoc opened 3 years ago

yadudoc commented 3 years ago

Launch funcx-endpoint following instructions here: https://github.com/funcx-faas/funcX/tree/forwarder_rearch_1/funcx_sdk/funcx/tests

Part of the test campaign: #318

yadudoc commented 3 years ago

@ZhuozhaoLi Do you think you could give this a try?

ZhuozhaoLi commented 3 years ago

Tests passed with the config below. Note: need to install anaconda by ourselves to install cryptography package.

from funcx_endpoint.endpoint.utils.config import Config
from funcx_endpoint.executors import HighThroughputExecutor
from parsl.providers import TorqueProvider
from parsl.launchers import AprunLauncher
from parsl.addresses import address_by_hostname

config = Config(
    executors=[
        HighThroughputExecutor(
            max_workers_per_node=1,
            worker_debug=False,
            address=address_by_hostname(),
            provider=TorqueProvider(
                queue='normal',
                launcher=AprunLauncher(overrides="-b -- bwpy-environ --"),
                scheduler_options='',  # string to prepend to #SBATCH blocks in the submit script to the scheduler
                worker_init='module load bwpy;source anaconda3/etc/profile.d/conda.sh;conda activate funcx_testing_py3.7',
                init_blocks=1,
                max_blocks=1,
                min_blocks=1,
                nodes_per_block=2,
                walltime='00:30:00'
            ),
        )

    ],
    # funcx_service_address='https://api.funcx.org/v1'
    funcx_service_address="http://k8s-dev.funcx.org/api/v1",
)