Question: Is it possible to decompress a Zip file?

[sparkyrider]

Is it possible to unzip a file from a zip file with Kumologica?

[bjq-dev]

There are few options:

1. Preferred one: We can prioritise zip node for release in couple of weeks if you can wait.
2. You can add zip module of your choice to the project's package.json file and it will be included into your deployment. You need to use function node to unzip it and AWS/Ephemeral node to access it.
3. Runtime is using adm-zip package, so you can use function node to unzip your payload and use AWS/Ephemeral node to write/read it. This is hacky, and may fail if we change this package in runtime in the future.

[sparkyrider]

I think option 1 would be best, however, I can use option 2 or 3 if needed.

Adding support for Compression/Decompression opens up a lot of amazing use cases for Big-data ingestion workflows, etc. when paired with AWS EFS.

Kumologica is amazing! I can't wait to see this tool become the next Unicorn company!

[ab73863]

Hi @sparkyrider , ZIP node is now available under "Add more nodes" catalogue. Please try and share your feedback. For a guidance on the node check the help section of the ZIP node editor.

For a quick reference I am providing a sample below

[{"id":"main.flow","type":"tab","label":"main","disabled":false,"info":""},{"id":"d9d038c9.9b47e8","type":"EventListener","z":"main.flow","name":"EventListener","provider":"aws","eventSource":"api","dynamodbOperation":"","apiMethod":"get","apiUrl":"/test","albMethod":"any","albUrl":"","bucketName":"","event":"s3:ObjectCreated:*","cognitoTrigger":"any","keventSource":"","kapiMethod":"any","kapiUrl":"","kcronexpression":"","zeventSource":"","zapiMethod":"any","zapiUrl":"","x":142.5,"y":160,"wires":[["a3db3c7a.43064"]],"caname":"event-handler","category":"general"},{"id":"a8084458.dc6388","type":"EventListener-End","z":"main.flow","name":"EventListener-End","statusCode":"200","headers":{},"payload":"","x":722.5,"y":160,"wires":[],"caname":"eventlistenerend","category":"general"},{"id":"c47f34ae.5de358","type":"ZIP","z":"main.flow","name":"Compress","payload":"msg.payload","filename":"","comment":"","operation":"compress","x":342.5,"y":160,"wires":[["85cd99b3.d48c98"]],"caname":"Zip","category":"File"},{"id":"85cd99b3.d48c98","type":"FTP","z":"main.flow","name":"FTP","operation":"Put","fileloc":"/kl/testkl2.zip","payload":"msg.payload","transfermode":"ASCII","username":"","password":"","hostname":"testing","port":21,"timeout":10000,"x":482.5,"y":160,"wires":[["a8084458.dc6388"]],"caname":"FTP","category":"File"},{"id":"a3db3c7a.43064","type":"ZIP","z":"main.flow","name":"Add the file","payload":"'Hello Kumologica zip'","filename":"kl.txt","comment":"testing","operation":"addfile","x":262.5,"y":160,"wires":[["c47f34ae.5de358"]],"caname":"Zip","category":"File"}]

Note : Install ZIP node and FTP node from "Add more nodes" to use the above given sample.

[ab73863]

Earlier sample was for zipping a text content and pushing to FTP location. The below is a sample where a zip having a text file is read from an FTP location and extracted using the ZIP node. The extracted content is printed using the logger node.

[{"id":"main.flow","type":"tab","label":"main","disabled":false,"info":""},{"id":"d9d038c9.9b47e8","type":"EventListener","z":"main.flow","name":"EventListener","provider":"aws","eventSource":"api","dynamodbOperation":"","apiMethod":"get","apiUrl":"/test","albMethod":"any","albUrl":"","bucketName":"","event":"s3:ObjectCreated:*","cognitoTrigger":"any","keventSource":"","kapiMethod":"any","kapiUrl":"","kcronexpression":"","zeventSource":"","zapiMethod":"any","zapiUrl":"","x":142.5,"y":160,"wires":[["85cd99b3.d48c98"]],"caname":"event-handler","category":"general"},{"id":"a8084458.dc6388","type":"EventListener-End","z":"main.flow","name":"EventListener-End","statusCode":"200","headers":{},"payload":"","x":722.5,"y":160,"wires":[],"caname":"eventlistenerend","category":"general"},{"id":"c47f34ae.5de358","type":"ZIP","z":"main.flow","name":"Extract","payload":"msg.payload","filename":"","comment":"","operation":"extract","x":382.5,"y":160,"wires":[["f0cb9c6c.e619c"]],"caname":"Zip","category":"File"},{"id":"85cd99b3.d48c98","type":"FTP","z":"main.flow","name":"FTP","operation":"Get","fileloc":"/kl/testkl2.zip","payload":"msg.payload","transfermode":"ASCII","username":"","password":"","hostname":"testing","port":21,"timeout":10000,"x":282.5,"y":160,"wires":[["c47f34ae.5de358"]],"caname":"FTP","category":"File"},{"id":"f0cb9c6c.e619c","type":"Function","z":"main.flow","name":"BufferToString","noerr":0,"query":"// Available objects in the context: msg, vars, env\n// Logging: node.log|info|warn|debug|error(\"message\")\n// Errors: throw new Error('message') \n\nlet buff = msg.payload[0].content\nmsg.payload = buff.toString();\n\nreturn msg;\n","x":502.5,"y":160,"wires":[["61c65717.b8b188"]],"caname":"function","category":"transformation"},{"id":"61c65717.b8b188","type":"Logger","z":"main.flow","name":"Logger","level":"INFO","message":"msg.payload","format":"string","headers":{},"x":622.5,"y":160,"wires":[["a8084458.dc6388"]],"caname":"logger","category":"logging"}]

[sparkyrider]

12 Aug 14:20:37 - [ warn] Communication send error: RangeError: Invalid string length

[sparkyrider]

For running a test case locally, is there a RAM limit setting that I can set or is it using w/e the local computer has available? I think the Compressed file is ~9MB Compressed, 60MB uncompressed.

[sparkyrider]

@ab73863 I was able to get it working on very small Zip files, however, even zip files uncompressed around 5-8mb fails with the same error. Is there a Debug Log somewhere I can check/share?

[ab73863]

@sparkyrider The error [ warn] Communication send error: RangeError: Invalid string length which you are seeing is due to a bug in the designer test tab. This is not an error due to any limitation of unzipping by the zip node. From runtime perspective the zip file that you have provided should be unzipped by the zip node and is available on memory. To check this you can actually try putting a logger after the ZIP node (post extraction) with the following on the Message : 'No of files extracted : ' & $count(msg.payload). This will print the number of files that got extracted and confirms the extraction was fine.

The output of extraction will be an array containing object having extracted file and its content as buffer. eg : [ {"filename" : "file1.png", "content" : [buffercontent] }, {"filename" : "file2.pdf", "content" : [buffercontent] }, {"filename" : "file3.txt", "content" : [buffercontent] } ]

The fix for the designer tab issue will be made available in our next release. And as I mentioned earlier you could still extract larger zip file on lambda without any issues. Will keep you posted on the fix update.

[ab73863]

@sparkyrider The patch for the designer covering the fix will be available next week.

[sparkyrider]

Zip node works great, looking forward to the Designer update. Thanks so much!