HermanFassett / youtube-scrape

Scrape YouTube searches (API)
MIT License
193 stars 96 forks source link

Is it possible to use the JSON generated in the demo directly on a personal page? #44

Open jetrotal opened 3 years ago

jetrotal commented 3 years ago

Hi, i'm kinda new to javascript developing for web and my question may sound stupid, but here we go:

I'm trying to use the scrapper to get a list of videos through this function:

        var CSEjson = "";
        var CSE = "";

        async function checkEngine(engine, i) {
            const response = await fetch(engine);
            CSEjson = await response.json();
            CSE = await "" + Object.keys(CSEjson)[0];

        }

But I keep getting 'Access-Control-Allow-Origin' errors that tell me that I don't have CORS permissions.

Is there something i can do on my end to make this work?

I tried deploying this repository on heroku to see if i can tinker with it.

but got stuck on these errors

2020-11-12T06:38:19.472262+00:00 app[web.1]: npm ERR! code ELIFECYCLE
2020-11-12T06:38:19.472547+00:00 app[web.1]: npm ERR! errno 1
2020-11-12T06:38:19.482877+00:00 app[web.1]: npm ERR! youtube-scrape@0.1.3 start: `node server.js`
2020-11-12T06:38:19.483063+00:00 app[web.1]: npm ERR! Exit status 1
2020-11-12T06:38:19.484248+00:00 app[web.1]: npm ERR! 
2020-11-12T06:38:19.484412+00:00 app[web.1]: npm ERR! Failed at the youtube-scrape@0.1.3 start script.
2020-11-12T06:38:19.484579+00:00 app[web.1]: npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

Once again, I have almost zero knowledge in NPM, or Node, or crossdomain comunication and these questions may not have a solution.

Thanks in advance!

HermanFassett commented 3 years ago

As is there's not much you can do if you're just working with browser side code. If you're making the request server-to-server then there should be no issues. A fix would be to add Access-Control-Allow-Origin: * header to youtube-scrape API so it will accept all sources, but I don't think that should be added to the codebase. If you can host it yourself, then you'll be able to specify your domain as an allowed source (if different) with that control header.

That's an odd error, have you tried setting the npm config production to false? (https://github.com/HermanFassett/youtube-scrape/issues/15#issuecomment-664034473)

jetrotal commented 3 years ago

As is there's not much you can do if you're just working with browser side code. If you're making the request server-to-server then there should be no issues. A fix would be to add Access-Control-Allow-Origin: * header to youtube-scrape API so it will accept all sources, but I don't think that should be added to the codebase. If you can host it yourself, then you'll be able to specify your domain as an allowed source (if different) with that control header.

That's an odd error, have you tried setting the npm config production to false? (#15 (comment))

I didn't put the entire log, sorry. here it is:

2020-11-12T15:41:19.213925+00:00 app[web.1]: 
2020-11-12T15:41:19.213952+00:00 app[web.1]: > youtube-scrape@0.1.3 start /app
2020-11-12T15:41:19.213952+00:00 app[web.1]: > node server.js
2020-11-12T15:41:19.213953+00:00 app[web.1]: 
2020-11-12T15:41:19.309259+00:00 app[web.1]: internal/modules/cjs/loader.js:834
2020-11-12T15:41:19.309261+00:00 app[web.1]:   throw err;
2020-11-12T15:41:19.309261+00:00 app[web.1]:   ^
2020-11-12T15:41:19.309262+00:00 app[web.1]: 
2020-11-12T15:41:19.309262+00:00 app[web.1]: Error: Cannot find module 'express'
2020-11-12T15:41:19.309262+00:00 app[web.1]: Require stack:
2020-11-12T15:41:19.309263+00:00 app[web.1]: - /app/server.js
2020-11-12T15:41:19.309264+00:00 app[web.1]:     at Function.Module._resolveFilename (internal/modules/cjs/loader.js:831:15)
2020-11-12T15:41:19.309264+00:00 app[web.1]:     at Function.Module._load (internal/modules/cjs/loader.js:687:27)
2020-11-12T15:41:19.309265+00:00 app[web.1]:     at Module.require (internal/modules/cjs/loader.js:903:19)
2020-11-12T15:41:19.309265+00:00 app[web.1]:     at require (internal/modules/cjs/helpers.js:74:18)
2020-11-12T15:41:19.309266+00:00 app[web.1]:     at Object.<anonymous> (/app/server.js:1:17)
2020-11-12T15:41:19.309266+00:00 app[web.1]:     at Module._compile (internal/modules/cjs/loader.js:1015:30)
2020-11-12T15:41:19.309266+00:00 app[web.1]:     at Object.Module._extensions..js (internal/modules/cjs/loader.js:1035:10)
2020-11-12T15:41:19.309267+00:00 app[web.1]:     at Module.load (internal/modules/cjs/loader.js:879:32)
2020-11-12T15:41:19.309267+00:00 app[web.1]:     at Function.Module._load (internal/modules/cjs/loader.js:724:14)
2020-11-12T15:41:19.309268+00:00 app[web.1]:     at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:60:12) {
2020-11-12T15:41:19.309268+00:00 app[web.1]:   code: 'MODULE_NOT_FOUND',
2020-11-12T15:41:19.309269+00:00 app[web.1]:   requireStack: [ '/app/server.js' ]
2020-11-12T15:41:19.309269+00:00 app[web.1]: }
2020-11-12T15:41:19.332086+00:00 app[web.1]: npm ERR! code ELIFECYCLE
2020-11-12T15:41:19.332669+00:00 app[web.1]: npm ERR! errno 1
2020-11-12T15:41:19.342503+00:00 app[web.1]: npm ERR! youtube-scrape@0.1.3 start: `node server.js`
2020-11-12T15:41:19.343073+00:00 app[web.1]: npm ERR! Exit status 1
2020-11-12T15:41:19.343462+00:00 app[web.1]: npm ERR! 
2020-11-12T15:41:19.343740+00:00 app[web.1]: npm ERR! Failed at the youtube-scrape@0.1.3 start script.
2020-11-12T15:41:19.344035+00:00 app[web.1]: npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
2020-11-12T15:41:19.355131+00:00 app[web.1]: 
2020-11-12T15:41:19.355693+00:00 app[web.1]: npm ERR! A complete log of this run can be found in:
2020-11-12T15:41:19.356014+00:00 app[web.1]: npm ERR!     /app/.npm/_logs/2020-11-12T15_41_19_345Z-debug.log

Looks like it doesn't have the express module and server.js it seems a lack of basic stuff from my side, i only tried to deploy and run the scrapper without knowing the basic setup to it.