Ignore media files smaller than problem

[Dean-Corso]

Hi again @rowrawer,

just wanna send a little feedback about a new tiny little issue I found when using / enable the function "Ignore media files smaller than" with any MegaByte X into. First problem is when I have enabled this option then I don't get any size informations of detections to see and just get a "-" under the size row. Why? I do detect the same file twice (with & without this otpion) and get two diffrent results back. Maybe you could fix this little issue.

The next problem I found when using this option is that not all custom extensions & Content-Type headers getting checked. I tried to detect a file of any website with the content type "application/octet-stream-m3u8" and your extension / AddOn failed to log the URL. In next try I did disable that option ""Ignore media files smaller than" and reload the website and then it did log the URL. There is any issue or more when having that option enabled and in use. Maybe you can have a look on that issue too. Otherwise I have this option disabled because I can not set it for custom extensions & Content-Type headers manually and just can set it for all files only. Anyway, just only few infos for you. Thanks.

[54ac]

Thanks for the unceasing feedback. The first issue doesn't really seem related, as the reported size has nothing to do with the filesize filter. Some files simply don't report their filesize and later you get the filesizes from cache (because you've loaded them twice). I'll look into the second issue in time for the next update.

[54ac]

The second issue should possibly be fixed in v2.10.17.

[Dean-Corso]

Hi @rowrawer,

thanks for the update but problem is still present if I have checked the option "Ignore media files smaller than: 0". Some types are not getting logged anymore but getting logged if I have it un-checked. I also still see some diffrent issues with the sizes logs. Some files shwoing sizes and some not but in pre version (2.10.12) its showing sizes for all (same files checked). All in all no big issues so far just reporting them to you to check if you want to know and fix it maybe.

PS: Got just one tiny request. Could you please add a small info / text about the version of your extension / addon when I open the options window? Just need this in case of using more than one diffrent versions of your tool at the same time and to know which options window I have open does belong to which versions etc. Thanks.

[54ac]

Having it set to 0 will probably lead to some weird behavior, the minimum really should be 1. Otherwise there isn't really anything in the code that would suggest what you're saying. Maybe something got changed a few versions back that caused this but I'm not really willing to put in the time to investigate this further. Maybe someone else can.

I will add a version tag in the next update.

[Dean-Corso]

Ok, maybe I just let this option unchecked so that the scan does work for sure. Otherwise the option is not good to use when you deal with much other custom stuff to check for.

Good to know you will add a version tag. 😃 Thank you.

[Dean-Corso]

Hey again @rowrawer,

just tried the latest version v2.10.19 and see that the size info is still not working good and now showing the size in many cases and just showing a "-" sign. At the same time I'am still using version 2.10.12 what does show the size of files which are not showing in the later and latest version. Why? Still not fixed? Just check this out and use both versions at the same time in Chrome based browser and check any websites (mp4, m4a etc not size info in latest and pre latest version till 2.10.12 what does show the info). Would be cool if you could fix that issue so that I can disable the old version. Thank you.

[54ac]

Hi, thanks for your continued feedback. I can't fix it because I don't know what causes the issue.

[Dean-Corso]

Hi @rowrawer,

thanks for the answer. Sounds bas that you can not find the problem between the older versions where it works and the newer where it no more works. 😢 So could you tell me what do you need to know to get the filesize and how you do check for it? What do you need for this? Whats the different between you data grabbing method and the browser itself where I can see the content-lenght filles paremeter? Like this...from browser...

HTTP/2 206 Partial Content
date: Wed, 11 Jan 2023 19:17:12 GMT
content-type: video/mp4
content-length: 383071

or

HTTP/2 206 Partial Content
date: Wed, 11 Jan 2023 19:22:13 GMT
content-type: audio/x-m4a
content-length: 2030593

...and your tools shows "-" for same one. Do you always check for that content-length paramter or just when you get status code 200 back?

[54ac]

Nothing that I know of changed between these versions related to content length and no one else reported this problem. I'm not sure how to approach it.

[Dean-Corso]

So if you nothing changed then it should also work. I think the problem started since you did added the new option "Ignore media files smaller than" so maybe you do check this again some more to find the reason. Otherwise just test this webpage... AudioFilesYouCanCheck ...click any of them. Here you tool will log this file as AAC type by itself but without size but the content-length are tere...

HTTP/2 206 Partial Content
date: Wed, 11 Jan 2023 21:02:23 GMT
content-type: audio/x-m4a
content-length: 1987375

...so you must have anything changed compared to the old version 2.10.12 what does work here. Don't wanna bother you about that but I would try to find it out you know.

PS: I think the most people just don't care about that size infos but I do. 😃

[54ac]

I believe you, I just can't figure out the correlation. I personally also don't really care about file sizes but I'll try to look into it again. If anyone else figures out the problem feel free to let me know.

[Dean-Corso]

Ok thank you @rowrawer. Maybe you will find the reason for this so that you can fix it and make it work again like in your older version 2.10.12. So the infos about the sizes are important for me to know the size of specific files I would like to handle go on or otherwise I have to request the size / content-length information manually which is of course a detour. Anyway, so just have a look on it if you have some time.

I think you should check the background.js files of the old / latest version. I did. Look... 2.10.12

const urlFilter = requestDetails => {
  let ext;
  let head;
  const url = new URL(requestDetails.url).pathname.toLowerCase(); // check file extension and see if the url matches

  ext = customExtPref === true && customSupported.ext?.some(fe => url.toLowerCase().includes("." + fe)) && customSupported;
  if (!ext) ext = supported.find(f => f.ext?.some(fe => url.toLowerCase().includes("." + fe)));
  const header = requestDetails.responseHeaders?.find(h => h.name.toLowerCase() === "content-type");

  if (header?.value) {
    // check content type header and see if it matches
    head = customCtPref === true && customSupported?.ct?.some(fe => header.value.toLowerCase().includes(fe.toLowerCase())) && customSupported;
    if (!head) head = supported.find(f => f.ct?.some(fe => header.value.toLowerCase() === fe.toLowerCase()));
  }

  const e = head || ext;

  if (e && !urlStorage.find(u => u.url === requestDetails.url) && // urlStorage because promises are too slow sometimes
  !queue.includes(requestDetails.requestId) && // queue in case urlStorage is also too slow
  requestDetails.tabId !== -1 && (!subtitlePref || subtitlePref && e.category !== "subtitles") && (!filePref || filePref && e.category !== "files") && (!manifestPref || manifestPref && e.category !== "stream") && (!blacklistPref || blacklistPref && !blacklistEntries?.some(entry => requestDetails.url?.toLowerCase().includes(entry.toLowerCase()) || (requestDetails.documentUrl || requestDetails.originUrl || requestDetails.initiator)?.toLowerCase().includes(entry.toLowerCase()) || header?.value?.toLowerCase().includes(entry.toLowerCase()) || e.type.toLowerCase().includes(entry.toLowerCase())))) {
    queue.push(requestDetails.requestId);
    requestDetails.type = e.type;
    requestDetails.category = e.category;
    addURL(requestDetails);
  }
};

2.10.19

const urlFilter = requestDetails => {
  let ext;
  let head;
  const url = new URL(requestDetails.url).pathname.toLowerCase();
  // check file extension and see if the url matches
  ext = customExtPref && customSupported.ext?.some(fe => url.toLowerCase().includes("." + fe)) && customSupported;
  if (!ext) ext = supported.find(f => f.ext?.some(fe => url.toLowerCase().includes("." + fe)));

  // depends which listener caught it
  requestDetails.headers = requestDetails.requestHeaders || requestDetails.responseHeaders;
  const headerCt = requestDetails.headers?.find(h => h.name.toLowerCase() === "content-type");
  if (headerCt?.value) {
    // check content type header and see if it matches
    head = customCtPref && customSupported?.ct?.some(fe => headerCt.value.toLowerCase().includes(fe.toLowerCase())) && customSupported;
    if (!head) head = supported.find(f => f.ct?.some(fe => headerCt.value.toLowerCase() === fe.toLowerCase()));
  }
  const headerSize = requestDetails.headers?.find(h => h.name.toLowerCase() === "content-length");
  const e = head || ext;
  if (e && !urlStorage.find(u => u.url === requestDetails.url) &&
  // urlStorage because promises are too slow sometimes
  !queue.includes(requestDetails.requestId) &&
  // queue in case urlStorage is also too slow
  requestDetails.tabId !== -1 && urlPrefValidator(e, requestDetails, headerSize, headerCt)) {
    queue.push(requestDetails.requestId);
    requestDetails.type = e.type;
    requestDetails.category = e.category;
    addURL(requestDetails);
  }
};

...do you see something? Must be some check in the urlPrefValidator function which fails to calc the right headerSize. So this code / function to handle content-length is not in the older version. I also see you do use 2 different struct path pointer names of "requestDetails" which is in the older version "requestDetails.responseHeaders?" and in newer version "requestDetails.headers?". Is it same path without to use response in name? Just asking. I think there is just any tiny wrong issue / check what the cause for this problem not showing the filesize anymore and showing this "-" instead in newer version. Maybe you see it above if so.

[54ac]

I'm going through the code now and I'm realizing what the issue is - I think that version had request listeners disabled and only relied on response listeners, and content length is only included in responses. Unfortunately having a bigger scope of detection is more important than file size, but I think it shouldn't be too hard to merge the headers from a matching request and response. I'll try to include this in the next version of the addon.

[54ac]

This should now finally be fixed in v2.11.0.

[Dean-Corso]

Hi @rowrawer,

thanks for the new update. I have test your latest version and its still not working like version 2.10.12. I still don't get the sizes for all files in latest version where version 2.10.12 does get & show the sizes. Sometimes it works sometimes not like a random. Also the flenames getting not always logged in latest version. Advantage version 2.10.12.

New problem. The icon does no more change to gray background when you did click on it so now all is gone and even does no more show the entry counter. If you did this on purpose than it was a bad idea so why did you change this?

Good thing. Seems that you fixed the problem to remove all entrys from the last session on exit or re-start just in case when I have un-checked the option "ignore pre sessions on launch" so then it removes really all from both tabs (current & pre) but If I have checked this option then all old entrys are still shown in the current tab. Somehow confusing. Anyway, now I keep this option un-checked so that all gets removed.

Just keep going so maybe you get those 2 buggy reading methods (filenames & sizes) fixed in future what would be nice of course and then I could also remove that old version from browser.

PS: If you got such trouble / scope reading the content-length from response then add another option in your app to enable to read filesizes or not. Maybe not all wanna know this info but I wanna know this info. Just add any 100% working method to read the content-length from response header IF this field is present.

[54ac]

thanks for the new update. I have test your latest version and its still not working like version 2.10.12. I still don't get the sizes for all files in latest version where version 2.10.12 does get & show the sizes. Sometimes it works sometimes not like a random. Also the flenames getting not always logged in latest version. Advantage version 2.10.12.

I'm sorry to hear that, but at this point I have no desire to keep spending my free time on this issue as it works fine enough for my needs and I simply don't know why it doesn't work as well as it used to for you. I might come back to it eventually, or I might remove the size column entirely, as it takes up space and has a limited use case (direct media file downloading, which is not the main focus of the addon). Feel free to stay on version 2.10.12. Of course anyone who can provide a better approach to the detection code is free to submit a pull request.

New problem. The icon does no more change to gray background when you did click on it so now all is gone and even does no more show the entry counter. If you did this on purpose than it was a bad idea so why did you change this?

Discussed in https://github.com/rowrawer/stream-detector/issues/171. I prefer this functionality.

[Dean-Corso]

Hi @rowrawer,

ah ok. As I said, just send you the results I got with your latest version. Still using the old & latest version together. 😃 Ok about that icon-counter change. I did prefer old style to see the entire count entrys. Now it does just reset on click. Before it was just going to change the background to gray color what was good and then it was changed to green again when something new was logged. Anyway.

[Dean-Corso]

Hi @rowrawer,

just found again a problem. Sometime your tools dosen't log any new datas anymore into the popup list. The icon does flash what means it got something new but when I open the popup window the current tab is empty and also in the all tabs it did not add new entrys. I think there is any refresh issue. Otherwise the problem is maybe that just too much entrys are logged. Did you add any size limit? After removing all entrys from all lists (all empty) it did work again.

Here my request again: Add a better clear function to clear everything on browser exit. Each time if I start my browser your app icon does flash for a moment and shows value X and then nothing. Otherwise, could you add a logging entry limit I can set?`I don't mean that "Show certain numbers" option I mean a real function what really does stop logging anything when a count of X was reached. Anyway, just can repeat myself to ask for a clean cleaning function. Thanks.

[54ac]

just found again a problem. Sometime your tools dosen't log any new datas anymore into the popup list. The icon does flash what means it got something new but when I open the popup window the current tab is empty and also in the all tabs it did not add new entrys. I think there is any refresh issue. Otherwise the problem is maybe that just too much entrys are logged. Did you add any size limit? After removing all entrys from all lists (all empty) it did work again.

It works fine for me. Maybe you had some sort of bugged entry for some reason. The addon wasn't really designed for heavy use or handling tons of entries, so I'd advise to clear your settings and entry lists every now and then. This also ties in to the detection algorithm again, but as I keep trying to tell you, I don't really plan on extensive bug hunting or adding intricate features to the addon for the time being.

Here my request again: Add a better clear function to clear everything on browser exit. Each time if I start my browser your app icon does flash for a moment and shows value X and then nothing. Otherwise, could you add a logging entry limit I can set?`I don't mean that "Show certain numbers" option I mean a real function what really does stop logging anything when a count of X was reached. Anyway, just can repeat myself to ask for a clean cleaning function. Thanks.

Doing anything on browser exit could be unreliable, everything is done on addon startup. Although as of v2.11.2 the clearing option will actually clear entries from previous sessions, as opposed to only hiding them. The icon still might flash, but that hardly sounds troublesome.

[Dean-Corso]

As I said, it would be nice if your tool would clear all entrys by itself on exit or re-start (real clear function). So for this you don't need to debug your own tool or anything. Just wish something what really works to clear all entrys. Just don't understand why its so complicated for you to add a function like that. Otherwise, maybe you could add another button in the popup window to clear all entrys with one click without to switch the tabs. I know you don't like the idea to add another button but I would prefer it to prevent switching from tab to tab (3) and delete all entrys by pressing the button X times you know. Thanks.

[54ac]

As mentioned in my previous message, the clear function in v2.11.2 will remove all entries from previous sessions on startup.

Just don't understand why its so complicated for you to add a function like that. Otherwise, maybe you could add another button in the popup window to clear all entrys with one click without to switch the tabs.

It's not complicated, I just don't want to change or add things, because I prefer the way they work now. Why would I change anything if I like it the way it is?

[Dean-Corso]

Because you are not the only one who is using your app! Otherwise fix the logging limit "Only show certain numbers.." what does just hide all entrys. Maybe you can change this just to log X entrys only and all older entrys getting removed automatically. Example: Lets say I just wanna let log the last 20 entrys only. On next new entry 21 the entry 20 should get removed to make free space for the latest entry and so on. In this case I would just get logged 20 last entrys only without to keep older entrys or just hide them etc you know. Just don't understand why you make such a uprising. 😃 What do you say about that idea? In this case you would also not increase the buffer and we as user would have quick access to the tabs too to switch what not happens when the buffer is filled with tons of entrys = long loading times.

[54ac]

Because you are not the only one who is using your app!

Yes, and the other users might not want the functionalities to be changed the way you want them to. Also, this is my personal addon which I am writing for my own use, as I've stated multiple times. Of course everyone's free to offer suggestions, report issues, fork the repo, submit pull requests, etc. but I'm only going to spend my free time on things that I personally decide are worth doing. In your case, I would suggest forking the addon and implementing all the hyper-specific changes yourself.

Otherwise fix the logging limit "Only show certain numbers.." what does just hide all entrys.

Again, there's nothing to "fix" here, it's working as intended. I will consider adding a checkbox that switches from hiding entries to deleting them, but no promises.

[Dean-Corso]

Yes, and thats the reason why you do close that thread! Just don't understand why you guys are always so narrow-minded and if you don't like to hear any other opinions then you just block or close a thread like this now. Sorry, but thats just ridiculous and childish behaviour from you. We can talk about it so long until you do recognize your fault.

PS: Yes of course, to expect a working "delete all entrys" & a "logging entry limit" are "hyper-specific changes" right! Just basic simple stuff not more or less and even this you don't get baked. Just a ascertainment no attack on you but I think even this you get into the wrong throat. Really a pity!