Closed Pythe1337N closed 2 years ago
Your CFClient callback is dropping the body in async scraper(options)
. I presume you tried to shortcut everything then realized this wouldn't work for the raw data methods...
Would be much better if cloud scraper had impl for streaming writes, but as is it does decompress the data to a buffer which could be written to disk...
Honestly I didn't put any work into investigating this issue. Realised it probably wouldn't work and just wanted to get a somewhat working version out there. If you have any suggestions on how to solve it, feel free to share!
Fix was simple IIRC, the aforementioned body just needs to not be dropped from the params list, also don't try to re-use the code you wrote everywhere. Just special case it with working code...
Suffice it to say I got it working and I fell down other rabbit holes. If I ever get back to my project perhaps I'll submit a patch.
I have added this to CFClient
async downloadFile(
id: string,
type: string,
url: string,
downloadDir: string = ""
) {
return new Promise(async (resolve, reject) => {
const fileName = `${id}.${type}`;
const filePath = path.resolve(downloadDir, fileName);
const writeStream = fs.createWriteStream(filePath);
request({
url,
method: "GET",
jar: this.cookies,
headers: this.headers,
})
.pipe(writeStream)
.on("finish", resolve)
.on("error", reject);
});
}
This to GarminConnect:
async downloadOriginalActivityData(
activity: any,
type: "zip" | "tcx" | "gpx",
dir: string
) {
const { activityId } = activity || {};
if (activityId) {
const url =
type === "zip"
? urls.originalFile(activityId)
: urls.exportFile(activityId, type);
return this.client.downloadFile(activityId, type, url, dir);
}
return Promise.reject();
}
And this to Urls:
const exportFile = (id: string, type: "tcx" | "gpx") =>
`${DOWNLOAD_SERVICE}/export/${type}/activity/${id}`;
It works. Thx for this lib!
If we can get this into a PR, I'll merge and release it.
In case @primozs gets to it before me, the only two things I'll say are:
1) There is an existing url - it might be nice to specify what type of download you get, the existing url gets you .fit files in a zip (what I needed) 2) Your approach uses request instead of cloudscraper, mine used the second. @Pythe1337N since I didn't pick the cloudscraper library I'm unfamiliar with exactly the reasoning behind that library, but to my understanding it handles things like captcha challenges whereas the straight request library does not, presumably this should be used preferentially?
@smaudet hi! please go ahead. Currently I don't have time.
I am observing failures on raw file downloads on 1.4.2. Sent #35 that fixed the issue for me.
Should be solved since v1.4.3
The code for doing this is disabled intentionally.
This ticket will keep track of this know issue.