Closed miroslavvojtus closed 5 years ago
Pretty sure this is related to this issue with the underlying request library that handles the http requests.
I've also have this issue where the first 16kb of the Azure Data Lake file are lost when using the Promise / Await code. For example, this code works just fine:
const localPath = "c:/some/place/nice.csv"
const accountName = "my-adl-account"
const remotePath = "/my/favorite/file.csv"
msRestAzure.interactiveLogin(function(err, credentials) {
const filesystemClient = new dataLakeClient.DataLakeStoreFileSystemClient(credentials);
let file = fs.createWriteStream(localPath )
file.on("open",() =>{
filesystemClient.fileSystem.open(accountName,remotePath,
(error,source) => {
if(error){return}
source.pipe(file)
})
})
})
but the promise based approach looses the first 16kb:
const localPath = "c:/some/place/nice.csv"
const accountName = "my-adl-account"
const remotePath = "/my/favorite/file.csv"
msRestAzure.interactiveLogin(function(err, credentials) {
const filesystemClient = new dataLakeClient.DataLakeStoreFileSystemClient(credentials);
let file = fs.createWriteStream(localPath )
file.on("open",() =>{
filesystemClient.fileSystem.open(accountName,remotePath)
.then(source => {
source.pipe(file)
})
})
})
To add to the discussion, I'm using Node version v8.9.3, with Typescript Version 3.0.1, my tsconfig includes the following:
{
"compilerOptions": {
"target": "es5",
"module": "commonjs",
"lib": ["ES2015"],
"strict": true,
"esModuleInterop": true
}
}
and my package-lock.json
file can be found here.
I am currently trying to integrate DataLake as a file storage with Node.js app. So far, if I omit typings issue which is already reported here, I have some problems with reading files.
I am using code similar to following one:
Problem is that the file written to the file system has missing first 16834 bytes (from its remote source). It sounds to me like first data emission already occurred and
stream.on("data", (d) => bufs.push(d));
receives only the rest of the file.The issue seems to be in promises and used streaming code.
When I tried to use callback variant without promise it seems to work fine:
Also, manual wrapping of returned stream seems to work: