yttrex extension and backend bug review for 2.5.0

vecna commented 2 years ago

yttrex extension

[x] send too many htmls of 'home', due to 'channel' leaf selector.

[x] scroll bar shouldn't appear in the popup (height:max ?)

⚠️ might be critical

[x] selector name was supposed to be a name and not a selector: but was it working anyway?

pict from background window -> network:

update: I checked the app.ts and the changes causes a change in the protocol that prevents leaveserv to work.

[x] fix researchTag, it should be saved from the popup

the function implemented by @djfm in #476, do not save anymore from the popup the server tag.


        function saveUpdate(payload) {
            bo.runtime.sendMessage({
                type: 'configUpdate',
                payload,
            }, (status) => {
                console.log('status confirmed', status);
            });
        }

from the popup console logs, the status is undefined. even if the element get read in the extension, isn't saved.

to test the bug try to write a research tag and see if stay persistent.

[ ] fix the researchTag completely, i can't understand why the changes aren't survived the merge 🤒 Note, the tag isn't sent even it preserved in the localStorage

yttrex backend

[x] experiments is saved also when there is no tag or any other signal

pict from backend:watch, it should not save experiments

[x] fix verbose error with a "not yet managed". in parserv:

 @trex:htmls:debug Processing element with nature [channel] +0ms
Error in channel: dissectorF is not a function TypeError: dissectorF is not a function
    at /home/oo/Dev/yttrex/platforms/yttrex/backend/lib/parser/parser.ts:50:20

[x] investigate memory leaks in parserv

 @trex:video:debug event 'YTD-COMPACT-VIDEO-RENDERER' at index 19 +1ms
  @trex:video:debug Video <History of The Universe Part 2: What Are the Five Ages of The Universe?> has 20 recommended (found 20, live {"false":20}) +1ms
  @trex:video:debug Views: { viewStr: '18 929&nbsp;vues', viewNumber: 18 } +3ms
  @trex:video:debug Like info: { likes: `379 clics sur "J'aime"`, dislikes: '' } +1ms
  lib:utils forceInteger convert 379 clics sur "J'aime" in 379 and then 379 +76ms
  @trex:htmls:debug Sleeping for 100 +130ms

<--- Last few GCs --->

[8123:0x57289d0]    59644 ms: Mark-sweep (reduce) 2041.2 (2080.2) -> 2040.2 (2080.4) MB, 992.9 / 0.0 ms  (average mu = 0.151, current mu = 0.009) allocation failure scavenge might not succeed
[8123:0x57289d0]    60632 ms: Mark-sweep (reduce) 2041.5 (2080.4) -> 2040.6 (2080.9) MB, 979.6 / 0.0 ms  (average mu = 0.082, current mu = 0.008) allocation failure scavenge might not succeed

<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0xb09980 node::Abort() [node]
 2: 0xa1c235 node::FatalError(char const*, char const*) [node]
 3: 0xcf784e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xcf7bc7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0xeaf465  [node]
 6: 0xeaff46  [node]
 7: 0xebe46e  [node]
 8: 0xebeeb0 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 9: 0xec1e2e v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
10: 0xe8336a v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node]
11: 0x11fc0b6 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [node]
12: 0x15f0b19  [node]
Aborted (core dumped)
npm ERR! Lifecycle script `parserv` failed with error: 
npm ERR! Error: command failed 
npm ERR!   in workspace: @yttrex/backend@2.4.6 
npm ERR!   at location: /home/oo/Dev/yttrex/platforms/yttrex/backend 
9633 ۞  ~/Dev/yttrex/platforms/yttrex/backend (chore/claudio-yt-ads) 
9633 ۞  ~/Dev/yttrex/platforms/yttrex/backend (chore/claudio-yt-ads) npm run parserv -- --minutesago 16000 -- --amount 1

I used the command line above, the memory leak happens after exactly one minute (there is something that runs every 60 seconds?)

[x] error with date-fns in parserv when parsing home

this has triggered once fixed the non-home-produced bug.

Error
    at toDate (/home/oo/Dev/yttrex/node_modules/date-fns/toDate/index.js:56:20)
    at differenceInMilliseconds (/home/oo/Dev/yttrex/node_modules/date-fns/differenceInMilliseconds/index.js:42:71)
    at differenceInSeconds (/home/oo/Dev/yttrex/node_modules/date-fns/differenceInSeconds/index.js:46:33)
    at /home/oo/Dev/yttrex/platforms/yttrex/backend/parsers/video.ts:285:41
    at arrayMap (/home/oo/Dev/yttrex/node_modules/lodash/lodash.js:653:23)
    at Function.map (/home/oo/Dev/yttrex/node_modules/lodash/lodash.js:9622:14)
    at Object.makeAbsolutePublicationTime (/home/oo/Dev/yttrex/platforms/yttrex/backend/parsers/video.ts:276:12)
    at process (/home/oo/Dev/yttrex/platforms/yttrex/backend/parsers/home.ts:394:35)
    at /home/oo/Dev/yttrex/platforms/yttrex/backend/lib/parser/parser.ts:50:20
    at step (/home/oo/Dev/yttrex/platforms/yttrex/backend/lib/parser/parser.ts:44:23)
Starting with v2.0.0-beta.1 date-fns doesn't accept strings as date arguments. Please use `parseISO` to parse strings. See: https://git.io/fjule

this is causing the recommendation to be published in the Epoch :smile_cat:

testing script with pm2 don't return expected values

[ ] from the personal page curl at the end of test.sh, the videos are missing

below there is the paste. it is the end of scripts/test.sh seems it hasn't processed (or collected?) the two youchoose videos. I added a 30 seconds delay because to let the schedule be sure to work. but hasn't worked either

EDdPY7jHXJVbPyGY2PBiMgDnfXuHBSHGcJH4Z4qUb9pt
Sleeping 30 seconds to let the parser process the data
{"supporter":{"_id":"6263180ca21f931f4c11fc03","publicKey":"EDdPY7jHXJVbPyGY2PBiMgDnfXuHBSHGcJH4Z4qUb9pt","version":"2.4.6.99","creationTime":"2022-04-22T21:03:08.602Z","lastActivity":"2022-04-22T21:03:08.602Z","p":"baklava-blueberry-celery","hereSince":"a few seconds"},"homes":[{"id":"82e81b551f98f16ea63cc0c510bb3218c5f8f490","savingTime":"2022-04-22T21:03:08.613Z","selected":[]}],"videos":[],"searches":[],"total":1,"stats":{"home":1},"ads":[],"request":{"amount":100,"skip":0,"when":"2022-04-22T21:09:05.636Z"}}[PM2] Applying action stopProcessId on app [all](ids: [ 0, 1, 2 ])

Protocol

[x] backend should not register an experiment when the experiment is empty,
[ ] test experiment feature with guardoni and guardoni-cli
[ ] verify if self-expiring mongodb document makes sense

this happens when navigating with a freshly .99 extension installed:

[INFO] 22:59:54 ts-node-dev ver. 1.1.8 (using ts-node ver. 9.1.1, typescript ver. 4.5.5)
  yttrex ઉ nconf loaded, using config/settings.json +0ms
  lib:mongo Initializing mongoUri with mongodb://localhost:27017/yttrex +0ms
  lib:debug password is set and it is not the default +0ms
  yttrex  Listening on http://0.0.0.0:9000 +34ms
  routes:experiments Experiment info {
  routes:experiments   href: 'https://www.youtube.com/',
  routes:experiments   testName: '2022-04-24T21:06:42.150Z',
  routes:experiments   publicKey: 'H314rD46MjdLErXAb7PztTqQuWxP4gHCutap4CCjdWxz'
  routes:experiments } +0ms
  routes:experiments Marked experiment as 'active' — {} +27ms
  api:debug {"responses":{"experimentChannel3":1},"requests":{"experimentChannel3":1}} +0ms
  routes:events Saved 1 htmls metadataId ["7902abbe5c3d46810cd481e50ac466ce4e2640b6"] +0ms

guardoni

[ ] make guardoni not ignore the --profile option

seems the profile requested isn't considered as default is loaded.

9716 ۞  ~/Dev/yttrex/platforms/guardoni (chore/claudio-yt-ads) dist/guardoni-cli-2.4.6-linux yt-auto 1 --profile test250 --backend http://localhost:9000 -v --evidenceTag test250-1
Configuration loaded from guardoni.config.json {
  profile: 'test250',
  backend: 'http://localhost:9000',
  evidenceTag: 'test250-1',
  'evidence-tag': 'test250-1',
  profileName: 'default',
  extensionDir: '../yttrex/extension/build',
  headless: false
}
  guardoni:debug Starting guardoni with config {
  guardoni:debug   profile: 'test250',
  guardoni:debug   backend: 'http://localhost:9000',
  guardoni:debug   evidenceTag: 'test250-1',
  guardoni:debug   'evidence-tag': 'test250-1',
  guardoni:debug   profileName: 'default',
  guardoni:debug   extensionDir: '../yttrex/extension/build',
  guardoni:debug   headless: false,
  guardoni:debug   verbose: true
  guardoni:debug } +0ms

[x] URL contacted by guardoni-cli in backend is invalid.

In the server you get:

api:debug URL not handled: /v3/directives/37384a9b7dff26184cdea226ad5666ca8cbbf456 +0ms

and from guardoni:

  guardoni:debug profile /home/oo/.guardoni/config/profiles/default/guardoni.json wrote {"udd":"/home/oo/.guardoni/config/profiles/default","newProfile":false,"profileName":"default","extensionDir":"../yttrex/extension/build","evidencetag":["no-tag-20952","no-tag-18881","no-tag-36418","no-tag-27505","test250-1"],"execount":5} +1ms
  guardoni:error Run error: APIError {
  guardoni:error   type: 'UnknownError',
  guardoni:error   message: 'Request failed with status code 404',
  guardoni:error   details: [],
  guardoni:error   name: 'APIError'
  guardoni:error } +0ms

[x] first output do not return date properly

9731 ۞  ~/Dev/yttrex/platforms/guardoni (chore/claudio-yt-ads) profile=test250 backend=http://localhost:9000 evidenceTag=test250-1BA dist/guardoni-cli-2.4.6-linux yt-register experiments/experiment-comparison.csv 

Register-csv succeeded: Experiment created successfully

Output values:

status: created
experimentId: 261c08120cf049a0c9ee9360015151a36e3523ad
since: undefined
9731 ۞  ~/Dev/yttrex/platforms/guardoni (chore/claudio-yt-ads) profile=test250 backend=http://localhost:9000 evidenceTag=test250-1BA dist/guardoni-cli-2.4.6-linux yt-register experiments/experiment-comparison.csv 

Register-csv succeeded: Experiment already available

Output values:

status: exist
experimentId: 261c08120cf049a0c9ee9360015151a36e3523ad
since: Mon Apr 25 2022 01:30:02 GMT+0200 (Central European Summer Time)
9731 ۞  ~/Dev/yttrex/platforms/guardoni (chore/claudio-yt-ads)

[x] guardoni-cli do not run exactly like the version not built by pkt

by testing the two different version, the build one fail to invoke a function from domainSpecific

compiled:

 guardoni:info — Loading https://www.youtube.com/watch?v=3laqegktOuQ (for 3000 ms) {
  guardoni:info   title: 'YouChoose: Customize your Recommendations',
  guardoni:info   url: 'https://www.youtube.com/watch?v=3laqegktOuQ',
  guardoni:info   urltag: 'youchoose-recommendations',
  guardoni:info   watchFor: 50000
  guardoni:info } +414ms
  guardoni:info Directive to URL https://www.youtube.com/watch?v=3laqegktOuQ, Loading delay 3000 (--load optional) +4s
Error in afterWait Passed function is not well-serializable! Error: Passed function is not well-serializable!
    at ExecutionContext._evaluateInternal (/snapshot/yttrex/platforms/guardoni/build/guardoni/cli.js)
    at ExecutionContext.evaluateHandle (/snapshot/yttrex/platforms/guardoni/build/guardoni/cli.js)
    at WaitTask.rerun (/snapshot/yttrex/platforms/guardoni/build/guardoni/cli.js)

non compiled:

  guardoni:info Directive to URL https://www.youtube.com/watch?v=ReVMTcRA-E4, Loading delay 3000 (--load optional) +5s
  guardoni:youtube afterWait status found to be: playing +8s
  guardoni:youtube DO NOT press [space] please, as the video is in state [playing] +12ms
  guardoni:youtube Entering watching loop (state playing) +3ms
https://www.youtube.com/watch?v=ReVMTcRA-E4 This video would be watched till the end
  guardoni:youtube Video status [ended] at check n#60 — closing loop +3m
  guardoni:info — Completed {
  guardoni:info   title: 'YouChoose.ai | Gain Control on your Recommendations!',
  guardoni:info   url: 'https://www.youtube.com/watch?v=ReVMTcRA-E4',
  guardoni:info   urltag: 'youchose-gain-Control',
  guardoni:info   watchFor: 'end'
  guardoni:info }

[x] guardoni-cli do not report output properly (publicKey missing, newProfile should have been true; see below the command given, in branch fix/yttrex-ads-ext

[ ] guardoni-cli in verbose mode do not say what actually was the problem, can network I/O be part of verbose logs?

vecna commented 2 years ago

Changes applied in these commits:

removed the channel-related leaves: they weren't used and they were mostly noise

vecna commented 2 years ago

edited to keep all the ☑️'s in the issue

vecna commented 2 years ago

@ascariandrea in home.ts this function is triggering always an error https://github.com/tracking-exposed/yttrex/blame/fix/yttrex-ads-ext/platforms/yttrex/backend/parsers/home.ts#L324, so it doesn't save the selected videos in homepage. which test might have prevented it? thanks!

fixed, referencing comment

vecna commented 2 years ago

In reference to the protocol, i was changing routes/experiments.jsto return a no experiment, I checked HandshakeBody and found how to specify { ignored: true }, and worked well!

vecna commented 2 years ago

how is it possible :P ?

ascariandrea commented 2 years ago

You may need to increase the heap memory for node with --max-old-space-size=4096

On Wed, 27 Apr 2022 at 11:39, Claudio Agosti @.***> wrote:

how is it possible :P ? [image: image] https://user-images.githubusercontent.com/89555/165489737-b620b44b-6a35-4357-bd75-3c77ace37212.png

— Reply to this email directly, view it on GitHub https://github.com/tracking-exposed/yttrex/issues/489#issuecomment-1110789292, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAOA3Z3WYIRCD7MFK6WH76TVHEDMHANCNFSM5TFYOC4A . You are receiving this because you were mentioned.Message ID: @.***>

tracking-exposed / trex