[Bug] Does not work on Windows 10 17763

[hugmouse]

When you download the release version. (https://github.com/McSib/e621_downloader/releases/download/1.4.3/e621_downloader.1.4.3.7z)

Error: Error: Error(Json(Error("expected value", line: 1, column: 1)))

tags.txt:

[artists]
fluff-kevlar

btw why it is ".txt" format if it is an ini-like format?

The other files have not been changed.

• [x] Figure out where the JSON error is coming from, whether it be from the parser, or from the web connecter.
• [x] Figure out the steps to take to cause the error on my system.
• [x] Locate the problem in code.
• [x] Fix bug.

[McSib]

There are a few questions I need to ask before I start looking into the issue.

• Do you get "Tags parsed" printed in the console before the JSON error is shown?
• Are you able to download a few images before this error is shown or does it happen before downloading even starts?
• Are you in safe mode?

I would also like you to present your config.json and your login.json (Show the login.json without your PasswordHash IF it contains your API key!)

Also, the reason the tag.txt file is a txt is for easy editing purposes. Notepad works immediately with it and allows people to easily add without doing any extra steps. It's more an ease-of-use choice. And even though the syntax is similar to an ini file, it is a custom language made with a custom parser. You can see the file here. The syntax takes inspiration from .ini syntax but has stricter rules towards how it is used.

[McSib]

I would like to add that I'm testing this software on Windows 10 1903 with OS build being 18362.535, so I'm not sure how Rust's compatibility works with older builds, especially since I'm using a few libraries, but I would assume that the build you're using wouldn't cause an issue with my software.

[hugmouse]

Do you get "Tags parsed" printed in the console before the JSON error is shown?

Here's the console output.

Are you able to download a few images before this error is shown or does it happen before downloading even starts?

Absolutely not. It didn't work right from the beginning.

I don't know if having a VPN can affect the operation of your program, but it shouldn't 🤔

Are you in safe mode?

It doesn't work in safe mode and without it either.

Configs

config.json

{
  "createDirectories": true,
  "downloadDirectory": "downloads/"
}

login.json

{
  "Username": "cummysh",
  "PasswordHash": "blah-blah-blah",
  "DownloadFavorites": false
}

[hugmouse]

Could you describe how to compile your project? It's just that if the release doesn't work, maybe the compiled project by myself will work for me personally 🤔

[McSib]

I want you to try something for me. Disable your VPN and try to use my program. The parser has an alias feature that sends requests to the e621 API, these calls may be rejected because of the VPN's IP. A lot of spam requests, attempted ddoses, etc, tends to happen through a VPN. I know that e621 has a few VPN servers blacklisted because of the listed reasons I gave. And having you compile the software yourself will not do much, since the only change made after the latest release was an update to the README.md file.

[McSib]

If your VPN ends up being the reason for the crash, be sure to run the software with it off. I can't do anything to help with that issue sadly, since it's a problem with the VPN provider themselves rather than my program.

[hugmouse]

I want you to try something for me. Disable your VPN and try to use my program. The parser has an alias feature that sends requests to the e621 API, these calls may be rejected because of the VPN's IP. A lot of spam requests, attempted ddoses, etc, tends to happen when a VPN is enabled. I know that e621 has a few VPN servers blacklisted because of the listed reasons I gave. And having you compile the software yourself will not do much, since the only change made after the latest release was an update to the README.md file.

I always have to use a VPN because a whole bunch of sites are blocked in my country. Without it, the result is expected:

Error: Error(Hyper(Error(Connect, Custom { kind: Other, error: Os { code: 10054, kind: ConnectionReset, message: "Удаленный хост принудительно разорвал существующее подключение." } })), "https://e621.net/user/blacklist.json?login=cummysh&password_hash=here_is_passowrd_hash")

Удаленный хост принудительно разорвал существующее подключение = An existing connection was forcibly closed by the remote host

By the way, e621 does not block my VPN.

[McSib]

Generally, e621 does not blacklist a VPN, however, API requests and/or calls are treated differently, since there are a lot of rules that go into it. If ill use of a VPN happens enough through one of their servers, e621 will blacklist it from using the API. I understand your situation about needing to use it though. From what I can see, the blacklist API call is being forcibly closed by the server itself.

Go to this URL with your login and password_hash put in, this should give you more information on why this error is occurring.

https://e621.net/user/blacklist.json?login=username_here&password_hash=here_is_passowrd_hash

[McSib]

The error is happening at this function within the web connector class. The only error that could be thrown is if the request is being rejected by the server, or the JSON supplied isn't what the program is expecting.

    pub fn grab_blacklist(&mut self) -> Result<(), Error> {
        if !self.login.is_empty() {
            let url = "https://e621.net/user/blacklist.json";
            // Here the request is sent and then turned into a container for the JSON data.
            // Since the data is stored as a Value, there can't be an error thrown from it being unable to parse the JSON correctly.
            let json: Value = self
                .client
                .get(url)
                .query(&[
                    ("login", self.login.username.as_str()),
                    ("password_hash", self.login.password_hash.as_str()),
                ])
                .send()?
                .json()?;
            // However, this field can throw an error if the field "blacklist" in the JSON doesn't exist.
            self.blacklist = json["blacklist"]
                .to_string()
                .trim_matches('\"')
                .replace("\\n", "\n");
        }

        Ok(())
    }

[hugmouse]

Generally, e621 does not blacklist a VPN, however, API requests and/or calls are treated differently, since there are a lot of rules that go into it. If ill use of a VPN happens enough through one of their servers, e621 will blacklist it from using the API. I understand your situation about needing to use it though. From what I can see, the blacklist API call is being forcibly closed by the server itself.

Go to this URL with your login and password_hash put in, this should give you more information on why this error is occurring.

https://e621.net/user/blacklist.json?login=username_here&password_hash=here_is_passowrd_hash

{"blacklist":"urine\nblood\ngynomorph"}

[McSib]

After running the tests on my program, using the latest release build, I had no issues occur. I'm a little confused about what is happening on my part, but I'm trying to locate the issue correctly. Something else that could be happening is that you are supplying the wrong login info. Are you using your API key or your password? The program uses the API key only. Is there a possible misspell in the login.json file, so the username or API key is invalid in some way?

If these are not the issue, then I will have to troubleshoot more. I even used my own VPN and had no issues occur, so this adds another layer of confusion for me.

[McSib]

Generally, e621 does not blacklist a VPN, however, API requests and/or calls are treated differently, since there are a lot of rules that go into it. If ill use of a VPN happens enough through one of their servers, e621 will blacklist it from using the API. I understand your situation about needing to use it though. From what I can see, the blacklist API call is being forcibly closed by the server itself. Go to this URL with your login and password_hash put in, this should give you more information on why this error is occurring. https://e621.net/user/blacklist.json?login=username_here&password_hash=here_is_passowrd_hash

{"blacklist":"urine\nblood\ngynomorph"}

Since you're making a proper connection with no issue, my only guess is something being supplied to my program with the login info is invalid or incorrect in some way. Or there is a bug in the code that I haven't experienced yet, and it's just now popping up. So I will have to look through it and see what's going wrong.

[McSib]

I'm going to compile a debug version of the program and give you a link, this will print a very verbose response when the error occurs.

[hugmouse]

I'm going to compile a debug version of the program and give you a link, this will print a very verbose response when the error occurs.

Okay, I'll wait, thanks!

[McSib]

Download this and try as if you were using the release version. If the error occurs, send me a screenshot of the entire error that's printed out. This will be a much slower version of the software, and almost quadruple the size since it holds debug info, so be patient with it.

https://www.dropbox.com/s/f7ufpt2z6vycaxk/e621_downloader%20Debug.7z?dl=0

[hugmouse]

Download this and try as if you were using the release version. If the error occurs, send me a screenshot of the entire error that's printed out. This will be a much slower version of the software, and almost quadruple the size since it holds debug info, so be patient with it.

https://www.dropbox.com/s/f7ufpt2z6vycaxk/e621_downloader%20Debug.7z?dl=0

(This is a debug version)

[McSib]

Let me try something real quick.

[McSib]

This version will print the received JSON data once the request is sent. If there is no error, close the program after taking another screenshot.

https://www.dropbox.com/s/zlbm183mjkgiesc/e621_downloader%20Debug.7z?dl=0

[hugmouse]

This version will print the received JSON data once the request is sent. If there is no error, close the program after taking another screenshot.

https://www.dropbox.com/s/zlbm183mjkgiesc/e621_downloader%20Debug.7z?dl=0

(debug v2.0 version)

[McSib]

Okay, as I figured the error would occur, time to get debugging on that then. I will send another version in a minute.

[McSib]

I'm hoping that this one will print the error with more details tied to it compared to what's been shown so far. This may also be a weird request, but run the software in cmd. I normally use Alacritty, but cmd will work best for debugging in this situation.

https://www.dropbox.com/s/0z2zkljrwem0drc/e621_downloader%20Debug.7z?dl=0

[hugmouse]

I'm hoping that this one will print the error with more details tied to it compared to what's been shown so far. This may also be a weird request, but run the software in cmd. I normally use Alacritty, but cmd will work best for debugging in this situation.

https://www.dropbox.com/s/0z2zkljrwem0drc/e621_downloader%20Debug.7z?dl=0

I'm going to install a wireshark and try to see what's wrong with the data that being sent.

[hugmouse]

And I'll try another windows system. Could be a problem with my Windows Enterprise (LTSC)

[McSib]

It seems the request is being sent okay, but the JSON is unable to be parsed. I'm going to code in a safeguard that prints information dynamically for this issue.

[hugmouse]

Tested your program on another system (AWS) and it works there. There seems to be something very wrong on my side 🤔

AWS Windows: 1809 17763.914

Proof

[McSib]

Try this on your enterprise system.

https://www.dropbox.com/s/s8x576rusueys77/e621_downloader%20Debug.7z?dl=0

[McSib]

It checks the status response code, prints the JSON directly, then trying to turn that JSON into usable data for the program to use.

[hugmouse]

Try this on your enterprise system.

https://www.dropbox.com/s/s8x576rusueys77/e621_downloader%20Debug.7z?dl=0

It's the same. Hell, I'm sorry about the weird bug.

[McSib]

I'm determined to fix this!

Okay, so I'm going to send the exe directly, because it's only showing the error, even though there is a lot more information that's supposed to be printed.

https://www.dropbox.com/s/mc8l1w80sti7im5/e621_downloader.exe?dl=0

[McSib]

The modified code looks like this now:

    pub fn grab_blacklist(&mut self) -> Result<(), Error> {
        if !self.login.is_empty() {
            let url = "https://e621.net/user/blacklist.json";
            let mut response: Response = self
                .client
                .get(url)
                .query(&[
                    ("login", self.login.username.as_str()),
                    ("password_hash", self.login.password_hash.as_str()),
                ])
                .send()
                .unwrap();
            if response.status() == 200 {
                let mut json_str = String::new();
                response.read_to_string(&mut json_str)?;
                println!("{}", json_str);
                let json = to_value(json_str.as_str())?;
                println!("{:#?}", json);
            } else {
                println!(
                    "Response is invalid, code response is {}",
                    response.status()
                );
            }
            //self.blacklist = json["blacklist"]
            //    .to_string()
            //    .trim_matches('\"')
            //    .replace("\\n", "\n");
        }

        Ok(())
    }

[hugmouse]

[McSib]

I really hope this isn't going to print right, because if so, then I'm about to face a much bigger problem...

Try this one: https://www.dropbox.com/s/mc8l1w80sti7im5/e621_downloader.exe?dl=0

[hugmouse]

Okay. I found a solution. Somehow, connecting to the VPN from Finland solves the problem, but if you connect to the VPN from St. Petersburg (and there are no blocked sites (and e621 working just fine)) - somehow the program stops working.

[hugmouse]

VPN Moscow:

VPN Finland:

[McSib]

As I feared, the JSON error is occurring in the parser itself, I'm going to set my VPN to Moscow and see if I can get an error.

[hugmouse]

My VPN provider has a P2P (file transfering) restriction on Russian servers, if this condition is important.

[McSib]

My provider doesn't even have Russia listed, but it does appear to be a problem with your provider. P2P shouldn't cause any problems with the software. I'm assuming that my program using the VPN from Russia is raising a flag, for some odd reason, and the server is cutting the connection off because anything could happen.

[McSib]

If I could get ahold of the status code that is sent, it would help nicely.

[McSib]

I will change my code slightly and send you that version. It will only print the code and then crash itself.

[McSib]

I am effectively probing the server for a response code, this will crash the program, but you should see a number along the 4xx or 5xx range.

https://www.dropbox.com/s/mc8l1w80sti7im5/e621_downloader.exe?dl=0

[hugmouse]

I'm hoping that this one will print the error with more details tied to it compared to what's been shown so far. This may also be a weird request, but run the software in cmd. I normally use Alacritty, but cmd will work best for debugging in this situation.

https://www.dropbox.com/s/0z2zkljrwem0drc/e621_downloader%20Debug.7z?dl=0

[hugmouse]

But there's nothing wrong in the browser 🤔

[McSib]

403 Forbidden code:

[McSib]

Your browser is okay because it contains a user-agent header confirming that it is certified and safe to use. I'm guessing some of my requests aren't pushing the user-agent header as they should.

[McSib]

Even then, this wouldn't make sense, as this code is thrown on sections where the user-agent is used correctly.

[McSib]

My only assumption is that the VPN running through my program just isn't trusted by the server, but the browser is. So it seems like the fix to this is just using a different location in the VPN, The problem that could be causing it is already fixed and has been since I started development on this program, so I don't know what could be causing this issue sadly.

[McSib]

The most I can say right now is I'll be working on security stuff for the next update, for right now, make do with the fix you found. Hopefully, the next release will put this bug to rest.

[hugmouse]

Thank you for developing this :)

[McSib]

No need to thank me, I made this program for myself, then for others. I use this quite often, so I make sure to always keep it up-to-date and working for all use cases. :3

I'm happy that you enjoy it, and hope you will continue using it in the future.

[McSib]

Hello, I have been doing research in trying to figure out this issue, looking into a lot of the issues that are ongoing with reqwest (the crate I use for establishing connections to e621), checking over the differences from Windows 10 Enterprise Edition to Windows 10 Pro Edition, and looking into how a VPN changes the way the web client connects to the server.

This is a compilation of what I know:

1. The first issue that could be happening is that your antivirus is breaking my software. Whether it be Avast, AVG, or Windows Defender, there is a chance that my program is detected as a false positive. A friend of mine tested my program recently and Avast raised a flag because my software wasn't in their database, so it would put it in their vault until it was scanned. There could be some interference with the software because your antivirus is preventing it from connecting to any URL.
2. My program relies on a crate called rust-native-tls for Windows and Mac, and OpenSSL for Linux. We'll focus on Windows for this case. The reqwest crate has certificates internally to free up the workload from programmers adding it themselves. More information on why these certificates are needed can be found here. Now, there could be a small chance that the program is failing the certification and returning a 404, but I must step in here and dismiss this. Since the crate relies on rust-native-tls for handling https certification, it will return an error explicitly stating that the certification failed. So my client is connecting to the e621 servers and it's getting a response, but the response is a 403 Forbidden.
3. The cache or cookies of the client is causing the response status to be a 403. This is a common error that can happen sometimes, and in those cases (with browsers), you can simply clear the cache and cookies and then the problem should be fixed. This isn't possible with my current setup, and it shouldn't be. A part of the next update will be ensuring that there is only one client being used, and that client should state and ensure that cookies are not kept.
4. What can be confirmed is that this is a server-side issue. Something with either the VPN, my program, or the client that my program is using is causing a 403 Forbidden code. Sounds like something on the client-side, right? Well, the server responding a code is showing that it did make a connection, and did confirm we were there, but it showed a 403 because of something wrong on the client-side. So I'm going to have to look into what else could be causing it.

I hope that the next release of my software fixes this. I'll be making quite a few changes to ensure this is as sound of a system it can be. I hope to keep in touch and ensure that you test the next release when it is released.