Allow task configuration overrides based on user input

[zvymazal]

Use Spawner.options_from_form for input values

[michalc]

Hi @zvymazal, thanks for the PR!

However, I think integration of options_form -> RunTask should now be possible with the latest API, using something like the below. It allows specification of overrides in RunTask, but also allows customisation of any the RunTask parameters. If you're feeling keen, happy to accept a modified PR that adds something like this to the README. (I would probably lean to the example not requiring a subclass of the spawner as well)

Michal

c.FargateSpawner.options_form = '''
<div class="form-group">
    <label for="resources">Choose suitable configuration for Notebook instance</label>
    <select name="resources" size="4" class="form-control">
      <option value="512_1024" selected>0.5 CPU & 1GB RAM</option>
      <option value="1024_4096">1 CPU & 4GB RAM</option>
      <option value="2048_16384">2 CPU & 16GB RAM</option>
      <option value="4096_30720">4 CPU & 30GB RAM</option>
    </select>
</div>
'''

c.FargateSpawner.get_run_task_args = lambda spawner: {
    'cluster': 'catalogue-dev-notebooks',
    'taskDefinition': 'catalogue-dev-local-fargate-dev:1',
    'overrides': {
        'taskRoleArn': 'arn:aws:iam::123456789012:role/notebook-task',
        'containerOverrides': [{
            'command': spawner.cmd + [f'--port={spawner.notebook_port}', '--config=notebook_config.py'],
            'environment': [
                {
                    'name': name,
                    'value': value,
                } for name, value in spawner.get_env().items()
            ],
            'name': 'jupyterhub-notebook',
        }],
        'cpu': spawner.user_options['resources'][0].split('_')[0],
        'memory': spawner.user_options['resources'][0].split('_')[1],
    },
    'count': 1,
    'launchType': 'FARGATE',
    'networkConfiguration': {
        'awsvpcConfiguration': {
            'assignPublicIp': 'DISABLED',
            'securityGroups': ['sg-00026fc201a4e374b'],
            'subnets':  ['subnet-01fc5f15ac710c012'],
        },
    },
}

[rhlarora84]

@michalc Thanks for the new version of the API. This will help supporting ECS CapacityProvider instead of the LaunchType which I believe will end up getting deprecated. Earlier I was using a monkey patch to run the task both using EC2 or Fargate.

My only ask (if possible) would be to make _run_task method part of the class as to open up the possibility of overriding. This would allow customizing the task details based on other components (ex. changing the docker image based on the logged in user but without an option form).

Happy to open a PR if you agree to the idea.

[michalc]

@rhlarora84 What would the overridden _run_task method do, in terms of specific calls to the ECS API?

[rhlarora84]

@rhlarora84 What would the overridden _run_task method do, in terms of specific calls to the ECS API?

Here is an example where by default I use EC2 Instance for launching Task but want to use FARGATE or FARGATE_SPOT as a CapacityProvider if no available EC2 has capacity to take on additional task. This is just an example. A better solution without overriding the fargatespawner would be to provide hook support.

    # An instance can run 4 tasks based on the CPU & Memory configuration. Autoscaling will scale so
    # if an instance is not immediately available, start the task using Fargate. Launch may be slow
    # but a user will not wait for provisioning
    ecs_client = boto3.client('ecs', region_name=region)
    instances = ecs_client.list_container_instances(cluster=os.getenv('ClusterName'), filter='runningTasksCount < 4',
                                                    status='ACTIVE')
    if not instances['containerInstanceArns']:
        capacity_provider = [{
            'capacityProvider': 'FARGATE'
        }]
        run_task_args['capacityProviderStrategy'] = capacity_provider
        run_task_args['platformVersion'] = '1.4.0'
        run_task_args['taskDefinition'] = run_task_args['taskDefinition'].replace('ec2', 'fargate')

    return await fargatespawner._make_ecs_request(logger, aws_endpoint, 'RunTask', run_task_args)

fargatespawner._run_task = _run_task```