romilbhardwaj
commented
2 years ago EDIT - see comments below.
Initial investigation:
TL;DR
☠☠☠Recursive ls operations on big buckets kills goofys☠☠☠
✅✅✅goofys is fine when ls operations are done on small dirs. Using generators for recursively going through files even in big buckets is ok. ✅✅✅
Setup
Test bucket
s3://fah-public-data-covid19-cryptic-pockets seems like a good candidate - ~I don't know how big it is because ls takes a very long time (good sign)~ Website says its 7TB. Has 4-5 level deep heirarchy of folder with many 5 MB files.
Cluster YAML
name: jupyter
resources:
cloud: aws
file_mounts:
/covid:
source: s3://fah-public-data-covid19-cryptic-pockets
mode: MOUNT
setup: |
pip install --upgrade pip
conda init bash
conda activate jupyter
conda create -n jupyter python=3.9 -y
conda activate jupyter
pip install jupyter
run: |
conda activate jupyter
jupyter notebook --port 8888Investigation
Wrote some code to parallel read files and print size:
# Read all files in a directory recursively in parallel
import os
from concurrent.futures import ThreadPoolExecutor
from concurrent.futures import as_completed
CAUSE_CRASH = False
def sizeof_fmt(num, suffix="B"):
for unit in ["", "Ki", "Mi", "Gi", "Ti", "Pi", "Ei", "Zi"]:
if abs(num) < 1024.0:
return f"{num:3.1f}{unit}{suffix}"
num /= 1024.0
return f"{num:.1f}Yi{suffix}"
def read_file(file):
print(file)
with open(file, 'rb') as f:
x=f.read()
#print("File: " + file)
#print("Size of file is :", sizeof_fmt(int(f.tell())))
def read_parallel(path):
files = []
print(f"Reading files in parallel {path}")
if CAUSE_CRASH:
ls_done = 0
for root, dirs, files in os.walk(path):
files = [os.path.join(root, file) for file in files]
ls_done += 1
if ls_done % 10 == 0:
print("ls done: ", ls_done)
print("Got files: ", len(files))
done = 0
for root, dirs, files in os.walk(path): # Uses a generator
with ThreadPoolExecutor(max_workers=64) as executor:
futures = [executor.submit(read_file, os.path.join(root, file)) for file in files]
for future in as_completed(futures):
done += 1
if done % 10 == 0:
print("Reads done: ", done)
print(future.result())
if __name__ == '__main__':
read_parallel('/covid')Turns out this works fine. Memory usage of goofys was stable around 37 MB for about 15 min of reading files (after which I stopped the process).
Afterwards, I added these lines before starting the os.walk loop:
for root, dirs, files in os.walk(path):
files = [os.path.join(root, file) for file in files]
print("Got files: ", len(files))This kills goofys because it's no longer a generator (there's something going on under the hood which translates this into a recursive stat syscall). The memory requirement keeps climbing in a bursty pattern (increases by ~1 GB in a few seconds, then stays there for a couple of min before increasing again). The command kept running for about 20 min and goofys was using about 6 GB in memory after which I ran out of patience. It never OOM'd, but I can see how it may happen for other workloads/buckets.
So what do we do?
Recursive ls is a fundamental challenge on these object stores (aws s3 ls also gets stuck for a long time, though it doesn't leak memory). Goofys probably inherits this problem from the fact that it's backed by an object store.
We could mention in our docs that we should avoid recursive ls operations. Maybe encourage the use of generators to do small ls ops. Any other ideas?
TODO
- Investigate how this interacts with common file reading patterns in ML (e.g. pytorch dataloader) and if we can do something to mitigate there
Michaelvll
concretevitamin
mraheja
github-actions[bot]
User report:
I was trying out the s3 file system mount. With some config like:
This worked ok initially and I was able to start a training job and run initialization of the dataloader. However half way through I’m starting to get errors like
And if I check the mounted folder it’s gone (also gone in df -h). When I first started the instance it was OK though.
from syslog, looks like goofys ran out of memory lol
using a g4dn.xlarge currently.
Usage pattern
there might be some concurrent operations after the directory listing steps too for data loading.
TODO: reproduce, investigate, mitigate?