search

ModClean / modclean

Remove unwanted files and directories from your node_modules folder
MIT License
398 stars 15 forks source link

Possible issue with default patterns and "license" keyword #36

Open mrapczynski opened 5 years ago

mrapczynski commented 5 years ago

I have been using Modclean ^3.0.0-beta.1 as part of a AWS Lambda packaging process to remove excess junk from the final .zip file prior to uploading to AWS for deployment.

However, at times, I've been chasing a weird problem where the .zip is uploaded with a damaged copy of the AWS Node.js SDK package. After stepping through each task on my build, I found out Modclean is the most likely culprint.

File Tree Excerpt Before:

-rw-rw-r-- 1 bamboo bamboo    8023 Oct 26  1985 kinesisvideomedia.d.ts
-rw-rw-r-- 1 bamboo bamboo     643 Oct 26  1985 kinesisvideomedia.js
-rw-rw-r-- 1 bamboo bamboo  149363 Oct 26  1985 kms.d.ts
-rw-rw-r-- 1 bamboo bamboo     541 Oct 26  1985 kms.js
-rw-rw-r-- 1 bamboo bamboo   86733 Oct 26  1985 lambda.d.ts
-rw-rw-r-- 1 bamboo bamboo     922 Oct 26  1985 lambda.js
-rw-rw-r-- 1 bamboo bamboo  121591 Oct 26  1985 lexmodelbuildingservice.d.ts
-rw-rw-r-- 1 bamboo bamboo     655 Oct 26  1985 lexmodelbuildingservice.js
-rw-rw-r-- 1 bamboo bamboo   29960 Oct 26  1985 lexruntime.d.ts
-rw-rw-r-- 1 bamboo bamboo     592 Oct 26  1985 lexruntime.js
-rw-rw-r-- 1 bamboo bamboo   30490 Oct 26  1985 licensemanager.d.ts
-rw-rw-r-- 1 bamboo bamboo     620 Oct 26  1985 licensemanager.js
-rw-rw-r-- 1 bamboo bamboo  237289 Oct 26  1985 lightsail.d.ts
-rw-rw-r-- 1 bamboo bamboo     583 Oct 26  1985 lightsail.js
-rw-rw-r-- 1 bamboo bamboo  124033 Oct 26  1985 machinelearning.d.ts
-rw-rw-r-- 1 bamboo bamboo     758 Oct 26  1985 machinelearning.js
-rw-rw-r-- 1 bamboo bamboo   14752 Oct 26  1985 macie.d.ts
-rw-rw-r-- 1 bamboo bamboo     555 Oct 26  1985 macie.js
-rw-rw-r-- 1 bamboo bamboo   15449 Oct 26  1985 marketplacecommerceanalytics.d.ts
-rw-rw-r-- 1 bamboo bamboo     716 Oct 26  1985 marketplacecommerceanalytics.js
-rw-rw-r-- 1 bamboo bamboo    6279 Oct 26  1985 marketplaceentitlementservice.d.ts
-rw-rw-r-- 1 bamboo bamboo     711 Oct 26  1985 marketplaceentitlementservice.js

Running Modclean as follows: node_modules/.bin/modclean --run --modules-dir=build/out/node_modules

Output:

MODCLEAN  Version 3.0.0-beta.1

Deleting files, please wait...
Deleting empty directories, please wait...

FILES/FOLDERS DELETED
    Total:    1713
    Skipped:  0
    Empty:    626

File Tree Excerpt After:

-rw-rw-r-- 1 bamboo bamboo    8023 Oct 26  1985 kinesisvideomedia.d.ts
-rw-rw-r-- 1 bamboo bamboo     643 Oct 26  1985 kinesisvideomedia.js
-rw-rw-r-- 1 bamboo bamboo  149363 Oct 26  1985 kms.d.ts
-rw-rw-r-- 1 bamboo bamboo     541 Oct 26  1985 kms.js
-rw-rw-r-- 1 bamboo bamboo   86733 Oct 26  1985 lambda.d.ts
-rw-rw-r-- 1 bamboo bamboo     922 Oct 26  1985 lambda.js
-rw-rw-r-- 1 bamboo bamboo  121591 Oct 26  1985 lexmodelbuildingservice.d.ts
-rw-rw-r-- 1 bamboo bamboo     655 Oct 26  1985 lexmodelbuildingservice.js
-rw-rw-r-- 1 bamboo bamboo   29960 Oct 26  1985 lexruntime.d.ts
-rw-rw-r-- 1 bamboo bamboo     592 Oct 26  1985 lexruntime.js
-rw-rw-r-- 1 bamboo bamboo  237289 Oct 26  1985 lightsail.d.ts
-rw-rw-r-- 1 bamboo bamboo     583 Oct 26  1985 lightsail.js
-rw-rw-r-- 1 bamboo bamboo  124033 Oct 26  1985 machinelearning.d.ts
-rw-rw-r-- 1 bamboo bamboo     758 Oct 26  1985 machinelearning.js
-rw-rw-r-- 1 bamboo bamboo   14752 Oct 26  1985 macie.d.ts
-rw-rw-r-- 1 bamboo bamboo     555 Oct 26  1985 macie.js
-rw-rw-r-- 1 bamboo bamboo   15449 Oct 26  1985 marketplacecommerceanalytics.d.ts
-rw-rw-r-- 1 bamboo bamboo     716 Oct 26  1985 marketplacecommerceanalytics.js
-rw-rw-r-- 1 bamboo bamboo    6279 Oct 26  1985 marketplaceentitlementservice.d.ts
-rw-rw-r-- 1 bamboo bamboo     711 Oct 26  1985 marketplaceentitlementservice.js

Note how licensemanager.js gets tossed, and unfortunately the AWS SDK fails to load because files are missing.

I looked at the default safe patterns, and it does not seem like the license term is treated with a wildcard, and yet it appears any file with the word license as part of the name get tossed.

Am I misunderstanding something?

saiteamf1 commented 5 years ago

modclean has default pattern license files to be cleaned. so file will be deleted.

herebebogans commented 4 years ago

I have a similar issue with src/languages/makefile.js from highlightjs being removed which breaks modules relying on it when running in AWS lambda.