qsniyg / maxurl

Finds larger/original versions of images and videos
https://qsniyg.github.io/maxurl/
Apache License 2.0
1.11k stars 69 forks source link

Direct links not permitted (imgtaxi_com) #593

Closed elvisef closed 3 years ago

elvisef commented 3 years ago

While using an addon in my Firefox browser, I see this - screenshot attached. All small images are not being replaced by their bigger versions. Instead, I see images "blocked" from displaying. The server is "imgtaxi.com".

I am using Debian 10.x and Windows 7.

2021-01-04_09-07-30

Can You look into it ?

Sample link: https://imgtaxi.com/img-5be361340943e.html

qsniyg commented 3 years ago

Can you see if https://github.com/qsniyg/maxurl/commit/1f8426c7c247604dc4d3a413605bc487ba5e9816 fixes it? I can't replicate the issue on my end, but I think on my end, it's because it's not embedded in a 3rd-party website. Theoretically, the commit should fix it, if the error is due to the referer header (which I would guess it is).

elvisef commented 3 years ago

I forgot to mention I didn't use a script for ex. greasemonkey etc., but a full Firefox addon, to replace content on visible page with more than one image. Please try to reproduce it using an addon, not a script to show image.

Hmm... it's a bit complicated to say, but it looks like Firefox and uBlock are blocking some content. I've just updated my script (yes, I installed MAXUrl a script for my greasemonkey before and now I've aupdated it with the newest file from Git) and even if uBlock is "disabled" with it's button, it block some content - the JS concolse in FF shows everything.

So I checked this directly, by pasting a blocked url into the browser and voilla - Firefox asks for permission to show content, because it's on a block list ("EasyList" or something) in uBlock.

So now, the "show image" script for MaxURL works for imgtaxi links, I see the image but.... look at my screenshot, some errors are still visible in the console.

firefox_2021-01-04_12-51-47

As You can see on the screenshot attached, a URL: https://imgtaxi.com/ea/fl.js is being blocked. We can see this in the console. But after pasting this link into the browser directly, the content is visible. It's weird.

What can be blocking it ? A Firefox itself ? Because in the incognito version I have only KeePass active and GreaseMonkey with one active script - MaxURL.

And as mentioned above, please try to reproduce my problem with the FF addon, which makes all thumbnails replacements not bigger pictures but pictures with texts, that are blocked.

qsniyg commented 3 years ago

As far as I can see, the errors in the screenshot are unrelated to image max url, and in fact the first one is good. The fl.js script is an anti-adblock script that ublock blocked. I'm not sure what the jquery error is from.

I tried with "replace images" on that page, but I still can't replicate the issue strangely (it works fine on my end)... Can you send your settings? (options -> export) Perhaps there's some combination of settings that causes the issue.

Do you experience the issue when clicking "replace images" on the imgtaxi page? Or on another website? It's possible the headers are not properly emulated..

Just out of curiosity though, can you try enabling "Use data URLs" under the Replace Images section? (in the Buttons category). It might work (though, if it works with the addon, that's indicative of another issue that needs to be fixed).

elvisef commented 3 years ago

Another try..

These is config exported from the addon (MaxURL FF addon): {"imu_enabled":true,"language":"en","check_updates":false,"check_update_interval":24,"check_update_notify":false,"dark_mode":false,"settings_tabs":true,"settings_alphabetical_order":false,"settings_visible_description":true,"settings_show_disabled":true,"settings_show_disabled_profiles":false,"settings_show_requirements":true,"advanced_options":false,"allow_browser_request":true,"retry_503_times":3,"retry_503_ms":2000,"use_blob_over_arraybuffer":false,"allow_live_settings_reload":true,"allow_remote":true,"disable_keybind_when_editing":true,"enable_gm_download":true,"gm_download_max":15,"enable_webextension_download":false,"write_to_clipboard":false,"redirect":true,"redirect_history":true,"redirect_extension":true,"canhead_get":true,"redirect_force_page":false,"redirect_infobox_url":false,"redirect_infobox_timeout":7,"print_imu_obj":false,"redirect_disable_for_responseheader":false,"redirect_to_no_infobox":false,"redirect_host_html":false,"mouseover":true,"mouseover_open_behavior":"popup","t2_mouseover_open_behavior":"popup","mouseover_trigger_behavior":"keyboard","mouseover_trigger_key":["shift","alt","i"],"mouseover_trigger_key_t2":[],"mouseover_trigger_delay":1,"mouseover_trigger_mouseover":false,"mouseover_trigger_enabledisable_toggle":"disable","mouseover_trigger_prevent_key":["shift"],"mouseover_close_behavior":"esc","mouseover_close_need_mouseout":true,"mouseover_jitter_threshold":30,"mouseover_cancel_popup_when_elout":true,"mouseover_cancel_popup_with_esc":true,"mouseover_cancel_popup_when_release":true,"mouseover_auto_close_popup":false,"mouseover_auto_close_popup_time":5,"mouseover_use_hold_key":true,"mouseover_hold_key":["i"],"mouseover_hold_position_center":false,"mouseover_hold_close_unhold":false,"mouseover_hold_unclickthrough":true,"mouseover_close_el_policy":"both","mouseover_close_click_outside":false,"mouseover_allow_partial":"media","mouseover_partial_avoid_head":false,"mouseover_use_blob_over_data":false,"mouseover_enable_notallowed":true,"mouseover_enable_notallowed_cant_load":true,"mouseover_notallowed_duration":300,"mouseover_minimum_size":20,"mouseover_exclude_backgroundimages":false,"mouseover_exclude_page_bg":true,"mouseover_exclude_imagemaps":true,"mouseover_only_links":false,"mouseover_linked_image":false,"mouseover_exclude_sameimage":false,"mouseover_exclude_imagetab":true,"mouseover_video_controls":false,"mouseover_video_controls_key":["c"],"mouseover_video_loop":true,"mouseover_video_autoloop_max":0,"mouseover_video_playpause_key":["space"],"mouseover_video_muted":false,"mouseover_video_mute_key":["m"],"mouseover_video_volume":100,"mouseover_video_volume_down_key":["9"],"mouseover_video_volume_up_key":["0"],"mouseover_video_volume_change_amt":5,"mouseover_video_resume_from_source":false,"mouseover_video_resume_if_different":false,"mouseover_video_pause_source":true,"mouseover_video_seek_amount":10,"mouseover_video_seek_left_key":["shift","left"],"mouseover_video_seek_right_key":["shift","right"],"mouseover_video_frame_prev_key":[","],"mouseover_video_frame_next_key":["."],"mouseover_video_framerate":25,"mouseover_video_speed_down_key":["["],"mouseover_video_speed_up_key":["]"],"mouseover_video_speed_amount":0.25,"mouseover_video_reset_speed_key":["backspace"],"mouseover_ui":true,"mouseover_ui_toggle_key":["u"],"mouseover_ui_opacity":80,"mouseover_ui_use_safe_glyphs":false,"mouseover_ui_imagesize":true,"mouseover_ui_zoomlevel":true,"mouseover_ui_filesize":false,"mouseover_ui_gallerycounter":true,"mouseover_ui_gallerymax":50,"mouseover_ui_gallerybtns":true,"mouseover_ui_closebtn":true,"mouseover_ui_optionsbtn":false,"mouseover_ui_downloadbtn":false,"mouseover_ui_rotationbtns":false,"mouseover_ui_caption":true,"mouseover_ui_wrap_caption":true,"mouseover_ui_caption_link_page":true,"mouseover_ui_link_underline":true,"mouseover_use_remote":false,"mouseover_zoom_behavior":"fit","mouseover_zoom_custom_percent":100,"mouseover_zoom_max_width":0,"mouseover_zoom_max_height":0,"mouseover_pan_behavior":"drag","mouseover_movement_inverted":true,"mouseover_drag_min":5,"mouseover_scrolly_behavior":"zoom","mouseover_scrollx_behavior":"gallery","mouseover_scrolly_video_behavior":"default","mouseover_scrolly_video_invert":false,"mouseover_scrollx_video_behavior":"default","scroll_override_page":false,"scroll_zoom_origin":"cursor","scroll_zoom_behavior":"fitfull","scroll_incremental_mult":1.25,"mouseover_move_with_cursor":false,"mouseover_move_within_page":true,"zoom_out_to_close":false,"scroll_past_gallery_end_to_close":false,"mouseover_position":"cursor","mouseover_prevent_cursor_overlap":true,"mouseover_add_link":true,"mouseover_add_video_link":false,"mouseover_click_image_close":false,"mouseover_click_video_close":false,"mouseover_download":false,"mouseover_hide_cursor":false,"mouseover_hide_cursor_after":0,"mouseover_mouse_inactivity_jitter":5,"mouseover_clickthrough":false,"mouseover_mask_ignore_clicks":false,"mouseover_links":false,"mouseover_only_valid_links":true,"mouseover_allow_iframe_el":false,"mouseover_allow_canvas_el":false,"mouseover_allow_svg_el":false,"mouseover_enable_gallery":true,"mouseover_gallery_cycle":false,"mouseover_gallery_prev_key":["left"],"mouseover_gallery_next_key":["right"],"mouseover_gallery_move_after_video":false,"mouseover_styles":"","mouseover_enable_fade":true,"mouseover_enable_zoom_effect":false,"mouseover_zoom_effect_move":false,"mouseover_fade_time":100,"mouseover_enable_mask_styles":false,"mouseover_mask_styles2":"background-color: rgba(0, 0, 0, 0.5)","mouseover_mask_fade_time":100,"mouseover_ui_styles":"","mouseover_wait_use_el":false,"mouseover_add_to_history":false,"mouseover_close_key":["esc"],"mouseover_download_key":[["s"],["ctrl","s"]],"mouseover_open_new_tab_key":["o"],"mouseover_open_bg_tab_key":["shift","o"],"mouseover_copy_link_key":["shift","c"],"mouseover_open_options_key":["p"],"mouseover_open_orig_page_key":["n"],"mouseover_rotate_left_key":["e"],"mouseover_rotate_right_key":["r"],"mouseover_flip_horizontal_key":["h"],"mouseover_flip_vertical_key":["v"],"mouseover_zoom_in_key":[["+"],["="],["shift","="]],"mouseover_zoom_out_key":[["-"]],"mouseover_zoom_full_key":["1"],"mouseover_zoom_fit_key":["2"],"mouseover_fullscreen_key":["f"],"mouseover_apply_blacklist":true,"apply_blacklist_host":false,"mouseover_matching_media_types":false,"mouseover_find_els_mode":"hybrid","popup_allow_cache":true,"popup_cache_duration":30,"popup_cache_itemlimit":20,"popup_cache_resume_video":true,"website_inject_imu":true,"website_image":true,"extension_contextmenu":true,"allow_video":true,"allow_dash_video":false,"allow_hls_video":false,"custom_xhr_for_lib":true,"hls_dash_use_max":true,"max_video_quality":null,"allow_watermark":false,"allow_smaller":false,"allow_possibly_different":false,"allow_possibly_broken":false,"allow_possibly_upscaled":true,"allow_thirdparty":false,"allow_apicalls":true,"allow_thirdparty_libs":true,"allow_thirdparty_code":false,"allow_bruteforce":true,"process_format":{},"deviantart_prefer_size":false,"deviantart_support_download":true,"ehentai_full_image":true,"imgur_filename":false,"imgur_source":false,"instagram_use_app_api":true,"instagram_dont_use_web":false,"instagram_prefer_video_quality":true,"instagram_gallery_postlink":false,"snapchat_orig_media":true,"tiktok_no_watermarks":false,"tiktok_thirdparty":null,"tumblr_api_key":"xrMpLM8n1eCeL3oRYSZG7CLALw6EHhYDdU6Wq5eD1PbMkld7Y1","bigimage_blacklist":"","bigimage_blacklist_engine":"glob","filename_format":"{author_username} {filename}","replaceimgs_enable_keybinding":true,"replaceimgs_keybinding":["shift","alt","r"],"replaceimgs_auto":false,"replaceimgs_replaceimgs":true,"replaceimgs_addlinks":false,"replaceimgs_replacelinks":false,"replaceimgs_links_newtab":false,"replaceimgs_usedata":false,"replaceimgs_wait_fullyloaded":true,"replaceimgs_totallimit":8,"replaceimgs_domainlimit":2,"replaceimgs_delay":0,"highlightimgs_enable_keybinding":false,"highlightimgs_keybinding":["shift","alt","h"],"highlightimgs_enable":false,"highlightimgs_auto":"never","highlightimgs_onlysupported":true,"highlightimgs_css":"outline: 4px solid yellow","last_update_check":1603137159364,"last_update_version":null,"last_update_url":null,"settings_version":7}

This is an example URL to test (sorry but I couldn't find anything else but girls :| ): http://niceupskirt.com/category/pictures/page/3

In this URL, if I do SHIFT + ALT + R - the magic is done, but:

a) images from the header "hot nurses with thong oops...." look good, but were not replaced 2021-01-04_22-20-49

b) images from the header "Argentinian girl with big ass festival public spy" are replaced by "direct link not permitted" firefox_2021-01-04_22-18-35

That's what I am trying to figure out, why images are not being replaced.

Moreover, for example images from "imagetwist" are replaced successfully on my side (still talking about the same URL as above). Only imgtaxi makes problem. I suppose imgadult, imgfrost and so on... very similar to imgtaxi, will also have problems on my side to be replaced using MaxURL addon.

So the fact is I am using this addon to make files bigger on URL, where there are more pictures as links from imgtaxi and other services, like the URL I pasted You above. I am not talkin now on greasemonkey's script for this.

And after doing such replacements, I use ImagePicka (FF addon) to download them all.

After all the text I wrote here... I tried "Use Data URLs" which de facto replaced images with biger ones, but I don't see direct links to images, just images still with HTML extension, but images are big, yeah... tried ImagePicka then to save them all but unfortunately too much RAM and addon failed ; [ So... what is that option, what does this means that it takes "Data URLs" ?

qsniyg commented 3 years ago

This is interesting... I'm able to replicate the issue with the userscript (with your settings), but not with the firefox addon.

Just to be clear, have you updated the addon with the git code? If you haven't, it will fail like the userscript version does. To update, download the repository in one way or another, go to about:debugging -> This Firefox -> Load Temporary Add-on, then navigate to the directory of the repository, and select manifest.json. Note that due to firefox restrictions, the updated version will be reverted once you close firefox.

By the way, the reason the userscript fails while the addon succeeds is because the userscript can only override headers for requests that it makes. So if it changes the source of an image, it cannot override those headers, as that is a browser request. Since addons have more permissions, it's able to change any headers. This is also why the "Data URLs" option works for the userscript. It requests the image manually (which allows the userscript to control the headers precisely), then sets the source url of the image to, quite literally, the entire image encoded in base64 (which explains the out of ram issue you mentioned).

The "hot nurses with thong" page worked fine for me when I did it on the page itself... I'd personally recommend only running replace images on a single page (i.e. on the least amount of image possible), instead of on a collection page like that, as it may trigger rate limits otherwise.

elvisef commented 3 years ago

More testing... Simple comparison - two URLs - two imgtaxi's:

1.http://niceupskirt.com/girl-with-bikini-opens-legs-spy-public - works - images are bigger. 2.http://niceupskirt.com/argentinian-girl-with-big-ass-festival-public-spy - does not work - images are still same default size

There are just a few images on each url page. No RAM issues, yeah.. but still weird thing, one works while the another doesn't.

I used the newest - git version FF addon of MAXurl of course (using Your advice of temporary adding it to FF - nice thing!).

So.. now the addon is newest. No blocking addons. Even Greasmonkey is all disabled now. There may be some differences between these two links, hmm... Of course these two pictures (one from 1st link, second from the 2nd link) have their own bigger size images available.

qsniyg commented 3 years ago

Are you sure you pasted the right URLs? The second one is imagetwist for me (and it works on my side...)

If the second url is indeed what you meant to paste, can you try opening the popup (shift+alt+i) over one of the images and see if it returns the original version? I'm curious what the issue is...

elvisef commented 3 years ago

Yes, sorry for that... Ive just updated the second link. First is working, second is not.

While using the shortcut key combination, the bigger size is working for ALL of them. So the popup shows the proper bigger size image.

Maybe a DOM structure differs a bit ? Or... some scripts are a bit different.

qsniyg commented 3 years ago

Ah! I think I understand what you're referring to now. Yes, you're right, the images aren't physically larger on the page, but the URLs have been replaced with larger versions:

screenshot of the image urls

This is because the image set the height/width properties in this one, whereas it doesn't in the other ones, so your first guess is absolutely right :)

I take it your use case with this is to scroll through the page and be able to see the larger images one by one, right? I could add an option to remove the height/width properties, but this would only work on a few sites (like this one), because of the complexities of CSS. It seems to me like there should be a better way of doing this...

In the meantime though, feel free to create a userstyle (using something like Stylus) with this, which should do the same thing:

.entry-content > div > a > img { width: auto; }

elvisef commented 3 years ago

Wow, yeah, I see it now. width and height, right. For me, making size bigger to see it bigger, is not an option. I mean, it is not crucial for me. I know it can be an option but for me what is important, I use ImagePicka addon, for FF, to download images from a page. So I need just to make sizes bigger - replace links etc.. I just didn't notice that this would be so simple "solution". The fact is it is not a problem, issue or whatever. But I think this may be a problem while using MAXurl addon for bigger amount of images on one page to replace. Just like in my case. So an option to force displaying in one size could be better : )

For example: 400x300 - and all "img" would be updated with this while replacing links. This would be a good option.

Anyway, You can also try adding a code to force update (if existing or not existing) attributes - width and height, of IMG element. And the problem will be solved I think.

qsniyg commented 3 years ago

Glad to know it works! I've added it now as a new option called "Remove size constraints" :)