search

awslabs / diagram-as-code

Diagram-as-code for AWS architecture.
Apache License 2.0
548 stars 35 forks source link

Unpacking Definition Files Results in "Too many open files" #136

Open phene opened 2 weeks ago

phene commented 2 weeks ago

Issue detail

Running awsdac on examples fails to unpack definitions due to "too many open files" error.

Mac M1 14.7 (23H124)

Reproduce process

$ awsdac examples/alb-ec2.yaml
FATA[0000] Cannot ExtractZipFile(/Users/phene/.cache/awsdac/ccecb6dfad3955969e4ce76bd853af4a-0546cce5532e24591c1c11360c79af62-AWS-Architecture-Icon-Decks.pptx.d6671530ea426cd7ff7fb00423bccb120bfc69df.zip/AWS-Architecture-Icons-Deck_For-Light-BG_06072024.pptx): Cannot create file with directory: Cannot create file(/Users/phene/.cache/awsdac/273c45b112d6d3f10470537d9bb0eda8-AWS-Architecture-Icons-Deck_For-Light-BG_06072024.pptx/ppt/media/image766.svg): open /Users/phene/.cache/awsdac/273c45b112d6d3f10470537d9bb0eda8-AWS-Architecture-Icons-Deck_For-Light-BG_06072024.pptx/ppt/media/image766.svg: too many open files

$ awsdac examples/alb-ec2.yaml
[Completed] AWS infrastructure diagram generated: output.png

The resulting file is missing icons. Deleting ~/.cache/awsdac and rerunning yields the same results.

output

phene commented 2 weeks ago

I was able to get around this by extracting the files manually from the pptx. I believe this issue is caused by all of the deferred file Close() operations leaving so many open file handles. It would probably be good to close some of them synchronously rather than waiting for the method's end.

ugwis commented 1 week ago

Thank you for reporting the issue! As you pointed out, defer fo.Close() did not work correctly and the file descriptor kept increasing. 

$ rm -rf ~/.cache/awsdac/ && strace ./awsdac examples/privatelink.yaml 2>&1 | grep -e open -e close
 ~ 
openat(AT_FDCWD, "/home/ugwis/.cache/awsdac/273c45b112d6d3f10470537d9bb0eda8-AWS-Architecture-Icons-Deck_For-Light-BG_06072024.pptx/ppt/media/image761.png", O_RDWR|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = 1017
openat(AT_FDCWD, "/home/ugwis/.cache/awsdac/273c45b112d6d3f10470537d9bb0eda8-AWS-Architecture-Icons-Deck_For-Light-BG_06072024.pptx/ppt/media/image762.svg", O_RDWR|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = 1018
openat(AT_FDCWD, "/home/ugwis/.cache/awsdac/273c45b112d6d3f10470537d9bb0eda8-AWS-Architecture-Icons-Deck_For-Light-BG_06072024.pptx/ppt/media/image763.png", O_RDWR|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = 1019
openat(AT_FDCWD, "/home/ugwis/.cache/awsdac/273c45b112d6d3f10470537d9bb0eda8-AWS-Architecture-Icons-Deck_For-Light-BG_06072024.pptx/ppt/media/image764.svg", O_RDWR|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = 1020
openat(AT_FDCWD, "/home/ugwis/.cache/awsdac/273c45b112d6d3f10470537d9bb0eda8-AWS-Architecture-Icons-Deck_For-Light-BG_06072024.pptx/ppt/media/image765.png", O_RDWR|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = 1021
openat(AT_FDCWD, "/home/ugwis/.cache/awsdac/273c45b112d6d3f10470537d9bb0eda8-AWS-Architecture-Icons-Deck_For-Light-BG_06072024.pptx/ppt/media/image766.svg", O_RDWR|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = 1022
openat(AT_FDCWD, "/home/ugwis/.cache/awsdac/273c45b112d6d3f10470537d9bb0eda8-AWS-Architecture-Icons-Deck_For-Light-BG_06072024.pptx/ppt/media/image767.png", O_RDWR|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = 1023
openat(AT_FDCWD, "/home/ugwis/.cache/awsdac/273c45b112d6d3f10470537d9bb0eda8-AWS-Architecture-Icons-Deck_For-Light-BG_06072024.pptx/ppt/media/image768.svg", O_RDWR|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = 1024
 ~

I've solved this issue by separating the function for writing to the file and believe this change prevents you from failing with too many open files error.

$ rm -rf ~/.cache/awsdac/ && strace ./awsdac examples/privatelink.yaml 2>&1 | grep -e open -e close
 ~ 
openat(AT_FDCWD, "/home/ugwis/.cache/awsdac/273c45b112d6d3f10470537d9bb0eda8-AWS-Architecture-Icons-Deck_For-Light-BG_06072024.pptx/ppt/media/image179.png", O_RDWR|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = 9
close(9)                                = 0
openat(AT_FDCWD, "/home/ugwis/.cache/awsdac/273c45b112d6d3f10470537d9bb0eda8-AWS-Architecture-Icons-Deck_For-Light-BG_06072024.pptx/ppt/media/image178.svg", O_RDWR|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = 9
close(9)                                = 0
openat(AT_FDCWD, "/home/ugwis/.cache/awsdac/273c45b112d6d3f10470537d9bb0eda8-AWS-Architecture-Icons-Deck_For-Light-BG_06072024.pptx/ppt/media/image177.png", O_RDWR|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = 9
close(9)                                = 0
openat(AT_FDCWD, "/home/ugwis/.cache/awsdac/273c45b112d6d3f10470537d9bb0eda8-AWS-Architecture-Icons-Deck_For-Light-BG_06072024.pptx/ppt/media/image176.svg", O_RDWR|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = 9
close(9)                                = 0
openat(AT_FDCWD, "/home/ugwis/.cache/awsdac/273c45b112d6d3f10470537d9bb0eda8-AWS-Architecture-Icons-Deck_For-Light-BG_06072024.pptx/ppt/media/image175.png", O_RDWR|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = 9
close(9)                                = 0
openat(AT_FDCWD, "/home/ugwis/.cache/awsdac/273c45b112d6d3f10470537d9bb0eda8-AWS-Architecture-Icons-Deck_For-Light-BG_06072024.pptx/ppt/media/image174.svg", O_RDWR|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = 9
close(7)                                = 0
 ~

This will be included in v0.21.7, but it has been already merged so you can build and test it yourself.