search

ZORALab / Hestia

One Peaceful Frontend+Backend Software Library Suite.
https://hestia.zoralab.com
Other
19 stars 1 forks source link

Hugo's ReadFile Function Crash Due to Non-streaming Nature #3

Closed hollowaykeanho closed 5 months ago

hollowaykeanho commented 2 years ago

Description

Please provide a short description of what you have encountered below.

When developing WASM embed feature function in https://github.com/ZORALab/Hestia/issues/1, it was discovered that Hugo does not offer 'streaming' nature of readFile function left alone byte restricted limits. Hence, when reading a large file (e.g. large WASM) beyond RAM capacity, on an OS without swap, it can crash the system entirely. This can be a security (stability) matter when any user tries to develop larger WASM while using hestiaHUGO Hugo theme.

Currently, there is no way to workaround this issue. It has to be solved on Hugo side.

Expected Behavior

Please specify the expected behavior of your usage below.

There is a streaming version of readFile that read a default or specified N number of data bytes at a time in a range loop format.

Current Behavior

Please specify the current behavior of software below.

No such feature available (Confimred in https://discourse.gohugo.io/t/streaming-version-of-readfile-function/40384/3).

Steps to Reproduce (COMPULSORY)

Please specify all the steps required to re-produce the problem.
  1. Get a small RAM (e.g. 4GB etc).
  2. Create a partial template.
  3. Read file larger than the designated RAM.
  4. Observe the crashing.

Attachments

Please drag and drop the necessary data files (e.g. screenshot, logs, etc)
below.
  1. Hugo data parsing functions - https://github.com/ZORALab/Hestia/blob/experimental/hestiaHUGO/layouts/partials/Hestia/functions/data/hugo/parse/__file
  2. Hestia filesystem data type - https://github.com/ZORALab/Hestia/blob/experimental/hestiaHUGO/layouts/partials/Hestia/functions/data/filesystem/ParseFile
hollowaykeanho commented 2 years ago

Communications initated to Hugo main development team. Pending reply.

hollowaykeanho commented 2 years ago

Hugo dev team acknowledged the submission. Pending response.

hollowaykeanho commented 2 years ago

Threat reduced to 2 file parsing functions listed above. WASM embed feature is temporarily disabled for the time being. Once hugo fixed the issue, we can launch it as "new feature" in the future.