Dynamically update ECS_RESERVED_MEMORY?

[dinvlad]

Hi Team,

Currently, the only way to "block out" memory external to ECS-scheduled containers (e.g. consumed by a TMPFS volume, or by a container scheduled directly through "docker run"), seems to be to specify a certain amount in advance via ECS_RESERVED_MEMORY. If this amount needs to change dynamically, we have to modify it in ecs.config, de-register that instance, and then register it again. That seems like a pretty disruptive change, especially that some tasks may still be running and would not normally be expected to fail or lost track of.

Or is there currently a way (or at least a workaround) to dynamically update the amount of memory not managed by ECS, ideally through an API call, that doesn't disrupt existing tasks on that instance?

Thank you

[samuelkarp]

@dinvlad There is not a current way to dynamically update the amount of ECS_RESERVED_MEMORY. I've marked this as a feature request to track that.

Just to help us in terms of determining priority, can you describe a bit of your use-case for modifying this value dynamically? That type of information helps us significantly when deciding the order in which we work on feature requests.

[dinvlad]

Our main use case was to set up TMPFS on the host and map its sub-folder as /dev/shm (POSIX shared memory) on each container, such that several containers running on the host would share that memory. This is important in some HPC workloads, where some applications load a single (yet large, e.g. 25GB) dataset into memory and then access it in read-only mode from multiple instances of the same application. Otherwise, we have to duplicate identical block of memory, which unnecessarily bloats memory requirement for each job (e.g. 16 cores x 25 GB = 400 GB at the least).

Modifying ECS_RESERVED_MEMORY option on the other hand would allow us to pre-allocate a certain amount of host memory for this purpose, and then mount its sub-folder to a group of containers without affecting other containers (as long as we don't decrease memory involved in their use). This would also be useful for task placement, where ECS would not schedule additional containers to a host where there is not enough memory, by counting both Reserved and Container memory.

One solution would be to use --shm-size option designed specifically for that, but then if we want to scale beyond 10 containers/task, we have to share this memory among ALL of the containers on the same instance (by specifying the less secure --ipc=host in Docker config to make it visible), meaning we have limited power to control access to it from containers belonging to different jobs. In addition, this option is also not directly supported by ECS at the moment (only through modifying Docker config).

One solution is obviously to just use threading, and run only a single copy of the application with multi-threading on the same container. Unfortunately, not all legacy software supports that (and we have no control over it, other than modifying third-party source code). In addition, that makes it less adaptable to the number of cores available on an instance (basically we have to specify in advance how many cores it should run with, and if no such instances are available, we are out of luck).

For the time being, we resorted to a zero-effort "hacky" solution that relies on the fact that the Linux kernel buffers the file into memory anyway, so next time another instance of the application requests the same file, it already resides in a memory buffer and will be just mapped to the second instance. This is not as performant as pure POSIX shared memory solution (presumably because some I/O is still spent on checking that the underlying FS blocks didn't change), but is still much faster than re-loading entirely from disk, and allows us to use instances with only a limited amount of RAM (e.g. 25GB) for all cores.

To put it in context, we are using this setup with the STAR aligner for genomic sequences: https://github.com/alexdobin/STAR

Thanks

[EldarSehayekZenity]

@samuelkarp I'd like to re-surface this issue as well, are there new advancements to share on the subject?

[samuelkarp]

I no longer work at Amazon.

[leroylim20]

+1