Open shamrin opened 11 months ago
I guess the fix is that we have to try loading the .env
file with multiple tries of encodings here: https://github.com/vitejs/vite/blob/5bb13aa6b299e7352b1db40914cb19cc66a52a19/packages/vite/src/node/env.ts#L37
And that would slightly hurt startup time, so I'm not really sure about fixing this. I think they should be strictly utf-8
, similar to how Vite reads any file in general, e.g. https://github.com/vitejs/vite/blob/5bb13aa6b299e7352b1db40914cb19cc66a52a19/packages/vite/src/node/plugins/loadFallback.ts#L8-L20
@bluwy Yes, trying multiple encodings seems unreliable and potentially slow. Enforcing utf-8
seems like a good solution. However, fsp.readFile
does not fail with this UTF-16 BOM file. Here is what happens:
import fsp from 'node:fs/promises'
try {
const r = await fsp.readFile('./.env.development.local', 'utf-8')
console.log(r)
console.log(Array.from(r).map(b => b.charCodeAt(0)))
} catch (e) {
console.log(e)
}
Output:
��VITE_VAR=1
[
65533, 65533, 86, 0, 73,
0, 84, 0, 69, 0,
95, 0, 86, 0, 65,
0, 82, 0, 61, 0,
49, 0, 13, 0, 10,
0
]
It looks like a fundamental limitation with readFile()
call. It simply ignores decoding errors and replaces weirdness with U+FFFD � replacement character
. Also keeps invisible '\0'
characters when dealing with UTF-16.
I think it could be solved if we decode with util.TextDecoder
:
import fsp from 'node:fs/promises'
import util from 'node:util'
const decoder = new util.TextDecoder('utf-8', {fatal: true})
const buffer = await fsp.readFile('./.env.development.local')
try {
console.log(decoder.decode(buffer))
} catch (e) {
console.log(e)
}
The above correctly fails with TypeError
for the broken .env
file:
TypeError: The encoded data was not valid for encoding utf-8
...
{
code: 'ERR_ENCODING_INVALID_ENCODED_DATA'
}
I'm still not sure if it's worth fixing, wouldn't that also affect perf? There's a lot of tools today that reads files with utf-8
and they would silently fail too, we're only fixing a portion of the issue.
@bluwy I believe it's worth fixing.
echo
command default encoding. Examples: dotenv, pipenv, phpdotenv.I haven't found measurable performance difference. Half of the time TextDecoder
was faster:
$ npm run benchmark
✓ test/encoding.bench.ts (2) 1309ms
name hz min max mean p75 p99 p995 p999 rme samples
· readFile 32,921.34 0.0100 12.5700 0.0304 0.0250 0.1150 0.1600 1.7350 ±8.41% 16461
· readFile with TextDecoder 36,706.53 0.0100 11.7250 0.0272 0.0200 0.1100 0.1450 1.4550 ±8.11% 18354 fastest
BENCH Summary
readFile with TextDecoder - test/encoding.bench.ts >
1.11x faster than readFile
https://stackblitz.com/edit/vitest-dev-vitest-oqhnfg?file=test%2Fencoding.bench.ts
FWIW fs.readFile(, 'utf8')
has a fast path (https://github.com/nodejs/node/pull/48658).
Even if we fixed every place in our code base, there'd be plenty of codes that we depend on that assumes the file is encoded in UTF-8. I don't think we can fix this.
@sapphi-red nice find! However, Vite does not supply "utf8"
argument when loading env files. I did a benchmark only to show there would be no measurable difference between current implementation (that silently ignores decoding errors) and a correct one (that tells users about incorrect encoding).
I would argue broken env handling on Windows is common enough to warrant a fix for this specific case (not general). I’ve posted some links in my previous comment.
I think It would be very nice if Vite would not fail silently and in a hard-to-debug manner.
How does Vite contribution flow work? Is it a good idea to get an approval from a maintainer first?
Describe the bug
Vite does not correctly handle
.env
files with BOM. For example, it happens if the file is created with Windowsecho
command:With this file
import.meta.env.VITE_VAR
expands toundefined
.Reproduction: https://stackblitz.com/edit/vitejs-vite-m3as94?file=main.js
Instead of a silent error I would expect one of two things:
'1'
(best).env.development.local
(better)Debugging
Outside of Windows, the file can be created with the Python script:
Related issue
https://github.com/motdotla/dotenv/issues/445
Reproduction
https://stackblitz.com/edit/vitejs-vite-m3as94?file=main.js
Steps to reproduce
No response
System Info
Used Package Manager
npm
Logs
No response
Validations