m4eba
commented
4 years ago Can you give me the url you try to download? Looks like the timeout problem caused by the eurosport backend, you also see that sometimes with the browser.
Open Elchava opened 4 years ago
m4eba
commented
4 years ago Can you give me the url you try to download? Looks like the timeout problem caused by the eurosport backend, you also see that sometimes with the browser.
Elchava
commented
4 years ago Hello,
Here is the url from source code
And the web url is
https://www.eurosportplayer.com/videos/eurosport-1-france/tour-de-france2014-5e-etape
Thanks !
m4eba
commented
4 years ago Sorry the link doesn't work for me, maybe geo-blocking?
But i have a different video that behaves the same i think. The page displays an error message and removes the player, that makes the script fail because it cannot find the title/date. I have now a different method the get the title, but for some reason it doesn't work in headless mode, the json url returns a 204 - no content.
m4eba
commented
4 years ago Got it working, please try the newest version.
Elchava
commented
4 years ago I have a new error, at the firts step, with the last version (like 10 or 15 loops with the same error) :
Error: expected data object not found 'included'
at video (/usr/local/bin/download.js:222:13)
at async /usr/local/bin/download.js:265:20
looking for www.eurosportplayer.com
Error: expected data object not found 'included'
at video (/usr/local/bin/download.js:222:13)
at runNextTicks (internal/process/task_queues.js:62:5)
at listOnTimeout (internal/timers.js:518:9)
at processTimers (internal/timers.js:492:7)
at async /usr/local/bin/download.js:265:20
looking for www.eurosportplayer.com
Error: expected data object not found 'included'
at video (/usr/local/bin/download.js:222:13)
at async /usr/local/bin/download.js:265:20
looking for www.eurosportplayer.com
Error: expected data object not found 'included'
at video (/usr/local/bin/download.js:222:13)
at async /usr/local/bin/download.js:265:20
looking for www.eurosportplayer.com
Error: expected data object not found 'included'
at video (/usr/local/bin/download.js:222:13)
at async /usr/local/bin/download.js:265:20
looking for www.eurosportplayer.com
Error: expected data object not found 'included'
at video (/usr/local/bin/download.js:222:13)
at async /usr/local/bin/download.js:265:20
looking for www.eurosportplayer.com
Error: expected data object not found 'included'
at video (/usr/local/bin/download.js:222:13)
at runMicrotasks (
m4eba
commented
4 years ago Added a fallback solution if the title is not found, the url will be used as filename. Give it a try.
Elchava
commented
4 years ago Now I have this message before putting my password and the captcha and after 5 loops the script fails :
TimeoutError: Timeout exceeded while waiting for event
at Timeout.
Elchava
commented
4 years ago Maybe timeout too short to put my ID and password ?
m4eba
commented
4 years ago If you use the -a option with a existing profile you don't need the login again. Start with -d and --trace command line parameter, that gives you more debug output and a browser window.
edit: you need to update first, i just added trace
Elchava
commented
4 years ago Thanks, here is the result :
args {
'login-timeout': 120000,
tmp: 'tmp',
out: './',
language: 'eng',
trace: true,
email: 'mail',
password: 'pass',
debug: true,
'user-data-dir': 'profile',
url: [ 'https://www.eurosportplayer.com' ]
}
testLoggin
goto https://auth.eurosportplayer.com/my-account
search "sign in" result -1
processing url https://www.eurosportplayer.com
video download https://www.eurosportplayer.com
looking for www.eurosportplayer.com
dataResp waitFor https://www.eurosportplayer.com/manifest.480eb9ea9b9eb064d2b06035c464f8b0.json
dataResp https://www.eurosportplayer.com/manifest.480eb9ea9b9eb064d2b06035c464f8b0.json
data {
icons: [
{
src: '/icon_512x512.604e2be3d212185553fba9b9413fd08e.png',
sizes: '512x512',
type: 'image/png'
},
{
src: '/icon_384x384.c0be9805ab5f96277b71df146406946c.png',
sizes: '384x384',
type: 'image/png'
},
{
src: '/icon_256x256.20031b64df8aac90c9437bc69a3b3880.png',
sizes: '256x256',
type: 'image/png'
},
{
src: '/icon_192x192.7e84a809c90c1eae56bc28d5d02eca34.png',
sizes: '192x192',
type: 'image/png'
},
{
src: '/icon_128x128.e3c5e7a1c1a15154169147e225ec53c9.png',
sizes: '128x128',
type: 'image/png'
},
{
src: '/icon_96x96.97e59ac52e12131ca720abf7e37f1422.png',
sizes: '96x96',
type: 'image/png'
},
{
src: '/icon_64x64.30a9d319d5735974d1aba73c8163568e.png',
sizes: '64x64',
type: 'image/png'
},
{
src: '/icon_32x32.8899f028ecd06d6805ea0c41991be771.png',
sizes: '32x32',
type: 'image/png'
},
{
src: '/icon_16x16.1856416143144af5fac164345037f4ec.png',
sizes: '16x16',
type: 'image/png'
}
],
name: 'Eurosport',
short_name: 'Eurosport',
display: 'standalone',
start_url: '.',
description: 'Eurosport an svod/avod streaming service',
theme_color: '#141b4d',
background_color: '#151726',
prefer_related_applications: true,
related_applications: [ { platform: 'play', id: 'com.eurosport.player' } ]
}
fallback title
waitFor m3u8
TimeoutError: Timeout exceeded while waiting for event
at Timeout.
m4eba
commented
4 years ago Oh, need to filter out email/password command line options. But you are already logged in with that profile in tmp, try to download with:
node download.js -a tmp --trace <url to download>
Elchava
commented
4 years ago With your last method, I obtain this result :
args { 'login-timeout': 30000, tmp: 'tmp', out: './', language: 'eng', trace: true, 'user-data-dir': 'tmp', url: [ 'https://www.eurosportplayer.com/videos/eurosport-2-france/cincinnati-1993-s-edberg-p-sampras' ] } testLoggin goto https://auth.eurosportplayer.com/my-account search "sign in" result 1126 login need email to login
And I have tried to put the timeout to 120000 with my email and password and here is the result:
args {
'login-timeout': 120000,
tmp: 'tmp',
out: './',
language: 'eng',
trace: true,
email: 'mail',
password: 'pass',
'user-data-dir': 'tmp',
url: [
'https://www.eurosportplayer.com/videos/eurosport-2-france/cincinnati-1993-s-edberg-p-sampras'
]
}
testLoggin
goto https://auth.eurosportplayer.com/my-account
search "sign in" result 1126
login
input email and password
click submit and wait with timeout 120000
unable to login TimeoutError: waiting for selector "button[class*="styles-authButton"]" failed: timeout 120000ms exceeded
at new WaitTask (/usr/local/bin/node_modules/puppeteer/lib/DOMWorld.js:549:28)
at DOMWorld._waitForSelectorOrXPath (/usr/local/bin/node_modules/puppeteer/lib/DOMWorld.js:478:22)
at DOMWorld.waitForSelector (/usr/local/bin/node_modules/puppeteer/lib/DOMWorld.js:432:17)
at Frame.waitForSelector (/usr/local/bin/node_modules/puppeteer/lib/FrameManager.js:627:47)
at Frame.
m4eba
commented
4 years ago Sorry misread the trace, the chrome profile folder should be 'profile'. So the command has to be
node download.js -a profile --trace <url to download>
I also removed email/password from the trace.
Elchava
commented
4 years ago Same error, if I don't put the email, I have : 'need email to login'
Otherwise, I get:
'login-timeout': 30000,
tmp: 'tmp',
out: './',
language: 'eng',
trace: true,
'user-data-dir': 'profile',
email: '',
password: '',
url: [
'https://www.eurosportplayer.com/videos/eurosport-2-france/cincinnati-1993-s-edberg-p-sampras'
]
}
testLoggin
goto https://auth.eurosportplayer.com/my-account
search "sign in" result 1127
login
input email and password
click submit and wait with timeout 30000
unable to login TimeoutError: waiting for selector "button[class*="styles-authButton"]" failed: timeout 30000ms exceeded
at new WaitTask (/usr/local/bin/node_modules/puppeteer/lib/DOMWorld.js:549:28)
at DOMWorld._waitForSelectorOrXPath (/usr/local/bin/node_modules/puppeteer/lib/DOMWorld.js:478:22)
at DOMWorld.waitForSelector (/usr/local/bin/node_modules/puppeteer/lib/DOMWorld.js:432:17)
at Frame.waitForSelector (/usr/local/bin/node_modules/puppeteer/lib/FrameManager.js:627:47)
at Frame.
m4eba
commented
4 years ago I added 2 more command line options for debugging: --har saves the requests and responses --save-html saves the html pages You also need to run 'npm install' again, i added a library. Add both options to the last command, it will generate har and html files, email and password should be removed. But to be sure mail me these files and the output: m4eba.0@gmail.com
Hello,
Could you help me with this issue for the second step ?
Error: Protocol error (Page.navigate): Cannot navigate to invalid URL at /usr/local/bin/node_modules/puppeteer/lib/Connection.js:183:56 at new Promise ()
at CDPSession.send (/usr/local/bin/node_modules/puppeteer/lib/Connection.js:182:12)
at navigate (/usr/local/bin/node_modules/puppeteer/lib/FrameManager.js:118:39)
at FrameManager.navigateFrame (/usr/local/bin/node_modules/puppeteer/lib/FrameManager.js:95:7)
at Frame.goto (/usr/local/bin/node_modules/puppeteer/lib/FrameManager.js:406:37)
at Frame. (/usr/local/bin/node_modules/puppeteer/lib/helper.js:112:23)
at Page.goto (/usr/local/bin/node_modules/puppeteer/lib/Page.js:672:49)
at Page. (/usr/local/bin/node_modules/puppeteer/lib/helper.js:112:23)
at video (/usr/local/bin/download.js:210:14)
at processTicksAndRejections (internal/process/task_queues.js:97:5)
at async /usr/local/bin/download.js:242:18
TypeError: Cannot read property 'm3u8' of null
at /usr/local/bin/download.js:249:18
at processTicksAndRejections (internal/process/task_queues.js:97:5)
Thanks a lot !