search

Otiel / BandcampDownloader

A Windows app used to download albums from Bandcamp.
Other
960 stars 76 forks source link

2024-09-12: Windows 7/8/8.1 are not supported anymore #256

Closed alabx closed 1 month ago

alabx commented 1 month ago

Will there be an update soon?

Otiel commented 1 month ago

It still works for me, and because you are the only one to report it, I think the issue comes probably from your side.

Can you browse https://bandcamp.com from your web browser?

Can you attach your log file to this issue?

alabx commented 1 month ago

Thanks for answering. It can't be from my side. I tested also on another computer with different internet source. It is the same, and in the past iw was good on both. It probably has to do with the last update of Bandcamp, because the problem started just 2-3 days after I noticed some changes in appearance on the webpage.

I can browse and listen without any problems. The log is: 15:49:03 Starting download... 15:49:03 Retrieving album data for (any link) 15:49:14 Could not retrieve data for (any link) - this line is red

Joey-2001 commented 1 month ago

I can confirm, that there are no problems with Bandcamp here. All downloads work as they should.

Otiel commented 1 month ago
darkdackel commented 1 month ago

I have the same problem: It ceased working/downloading a few days ago. It might be an issue with -NET framework. You might need Vs. 4.8.1 for windows or higher (which does not support WIN 7 anymore!).

Otiel commented 1 month ago

Thanks for chiming in. For everyone having this issue, can you please attach your log file?

BobFosko commented 1 month ago

I can confirm that it still works on my Win10 machine, but not on my W7 box. Latest version, etc. I did a .net framework update on the W7 box a while ago, that might be the issue? I have a old Visual Studio 2019 running on the W7 box, so could debug for you if you like. This is what I currently get as errormessage in the IDE, no breakpoints set, just running the app and trying to download something:

'BandcampDownloader.exe' (CLR v4.0.30319: BandcampDownloader.exe): Loaded 'BandcampDownloader.resources'. Module was built without symbols.

[...]

The thread 0x2e8c has exited with code 0 (0x0). Exception thrown: 'System.Net.WebException' in mscorlib.dll The program '[1344] BandcampDownloader.exe' has exited with code 0 (0x0).

Let me know if I can help out.

BobFosko commented 1 month ago

So I browsed arounf the code a bit and started debugging and ended up at private async Task<List> GetAlbumsAsync(List urls) where htmlCode = await webClient.DownloadStringTaskAsync(url); raises an exception, so it looks like the .NET W7 webclient wrapper no longer works with the code.

Again, Let me know if I can help out.

AudioEuphoria commented 1 month ago

Having the same issue on Win7, it started a few days ago.

BandcampDownloader.log

Stuck with this old rig for another year, I never realized how dependent I've become on this program until this week!

Otiel commented 1 month ago

I did a .net framework update on the W7 box a while ago, that might be the issue?

@BobFosko What .NET Framework version did you install? Was it proposed via Windows Update?

alabx commented 1 month ago
  • Are you using a VPN? No.
    • What country are your from? Poland.
    • Can you access https://t4.bcbits.com from your web browser (you should have an "Error 403 Forbidden")? Yes.
    • Please attach your entire log file (BandcampDownloader.log, next to BandcampDownloader.exe), not just copy/paste some lines of the log. At the moment I can't because I am on another device, but as I remember there was no more details from what I posted above. I also have Windows 7 and NET framework 4.7.2, but I don't know what it has to do with the program that is the same.
Otiel commented 1 month ago

htmlCode = await webClient.DownloadStringTaskAsync(url); raises an exception, so it looks like the .NET W7 webclient wrapper no longer works with the code.

@BobFosko Is there a stack trace or a more detailed message with the exception?

BobFosko commented 1 month ago

I did a .net framework update on the W7 box a while ago, that might be the issue?

@BobFosko What .NET Framework version did you install? Was it proposed via Windows Update?

No, it's on an old W7 box, which obviously does not get any updates anymore. I was trying to load a project that required a later framework of .NET that was installed, and after a few failed attempts found out that VS2019 did not support that version, so ended up switching to VS2022 on a W10 machine. But it might be the install of "MS Windows Desktop Runtime - 8.0.8" that messed things up on the W7 side.

BobFosko commented 1 month ago

htmlCode = await webClient.DownloadStringTaskAsync(url); raises an exception, so it looks like the .NET W7 webclient wrapper no longer works with the code.

@BobFosko Is there a stack trace or a more detailed message with the exception?

I'll try and set it up and have a look later.

BobFosko commented 1 month ago

Here's the stack trace after I hit the catched exception in webClient.DownloadStringTaskAsync(url);

BandcampDownloader.exe!BandcampDownloader.LogArgs.LogArgs(string message, BandcampDownloader.LogType logType) Line 9    C#
BandcampDownloader.exe!BandcampDownloader.DownloadManager.GetAlbumsAsync(System.Collections.Generic.List<string> urls) Line 392 C#
[Resuming Async Method] 
mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx)   Unknown
mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx)   Unknown
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.MoveNextRunner.Run()    Unknown
WindowsBase.dll!System.Windows.Threading.ExceptionWrapper.InternalRealCall(System.Delegate callback, object args, int numArgs)  Unknown
WindowsBase.dll!System.Windows.Threading.ExceptionWrapper.TryCatchWhen(object source, System.Delegate callback, object args, int numArgs, System.Delegate catchHandler) Unknown
WindowsBase.dll!System.Windows.Threading.DispatcherOperation.InvokeImpl()   Unknown
WindowsBase.dll!MS.Internal.CulturePreservingExecutionContext.CallbackWrapper(object obj)   Unknown
mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx)   Unknown
mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx)   Unknown
mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) Unknown
WindowsBase.dll!MS.Internal.CulturePreservingExecutionContext.Run(MS.Internal.CulturePreservingExecutionContext executionContext, System.Threading.ContextCallback callback, object state)  Unknown
WindowsBase.dll!System.Windows.Threading.DispatcherOperation.Invoke()   Unknown
WindowsBase.dll!System.Windows.Threading.Dispatcher.ProcessQueue()  Unknown
WindowsBase.dll!System.Windows.Threading.Dispatcher.WndProcHook(System.IntPtr hwnd, int msg, System.IntPtr wParam, System.IntPtr lParam, ref bool handled)  Unknown
WindowsBase.dll!MS.Win32.HwndWrapper.WndProc(System.IntPtr hwnd, int msg, System.IntPtr wParam, System.IntPtr lParam, ref bool handled) Unknown
WindowsBase.dll!MS.Win32.HwndSubclass.DispatcherCallbackOperation(object o) Unknown
WindowsBase.dll!System.Windows.Threading.ExceptionWrapper.InternalRealCall(System.Delegate callback, object args, int numArgs)  Unknown
WindowsBase.dll!System.Windows.Threading.ExceptionWrapper.TryCatchWhen(object source, System.Delegate callback, object args, int numArgs, System.Delegate catchHandler) Unknown
WindowsBase.dll!System.Windows.Threading.Dispatcher.LegacyInvokeImpl(System.Windows.Threading.DispatcherPriority priority, System.TimeSpan timeout, System.Delegate method, object args, int numArgs)   Unknown
WindowsBase.dll!MS.Win32.HwndSubclass.SubclassWndProc(System.IntPtr hwnd, int msg, System.IntPtr wParam, System.IntPtr lParam)  Unknown
[Native to Managed Transition]  
[Managed to Native Transition]  
WindowsBase.dll!System.Windows.Threading.Dispatcher.PushFrameImpl(System.Windows.Threading.DispatcherFrame frame)   Unknown
PresentationFramework.dll!System.Windows.Application.RunDispatcher(object ignore)   Unknown
PresentationFramework.dll!System.Windows.Application.RunInternal(System.Windows.Window window)  Unknown
BandcampDownloader.exe!BandcampDownloader.App.Main()    Unknown
[Async Call Stack]  
[Async] BandcampDownloader.exe!BandcampDownloader.DownloadManager.FetchUrlsAsync() Line 85  C#
[Async] BandcampDownloader.exe!BandcampDownloader.WindowMain.StartDownloadAsync() Line 225  C#
Otiel commented 1 month ago

Thanks. No message on the exception?

BobFosko commented 1 month ago

Nope. It's pretty useless.

alabx commented 1 month ago

I just want to add that on Windows 8.1 with NET Framework 4.8 is the same problem. Do you have an idea why it is like that and is it possible to be repaired?

Otiel commented 1 month ago

I've spent a few hours last night mounting a Windows 7 virtual machine. I could reproduce the problem but I still have no definite idea on what causes this issue.

This is clearly platform dependent as there is no problem on Windows 10. I have a fear that it is due to some certificates root that could have expire on unsupported Operating Systems like Windows 7 or Windows 8.1, but with no certainty. This would make all downloads fail when using the .NET Framework on those OS.

[Out of topic] As a reminder, Windows 7 and Windows 8.1 are not supported anymore (you can check the dates of the end of support on https://endoflife.date/windows ), meaning that no security updates are pushed through Windows Update on your machines. This means that your machines are vulnerable, I would strongly suggest you to upgrade to Windows 10/11. [/Out of topic]

Anyway, I'll try to make more tests tonight if I find the time. Sorry I don't have a better answer yet, I have little time nowadays. 😔

alabx commented 1 month ago

I hope you will fix the problem. I am no programmer, but maybe you can integrate needed certificates into your program. If it helps, it stopped working on 12.09.2024. A day before it worked.

alabx commented 1 month ago

Do you have any idea how to repair this useful program? I don't want to rush you, but I'm seriously concerned about the lack of information.

Otiel commented 1 month ago

No luck yet, but I'm trying to find a solution.

Tindalos commented 1 month ago

BandcampDownloader.log

Just recently started having this problem on 7 too. Haven't updated a thing recently. Was fine at the start of the month.

Greatly appreciate your diligence Otiel, I know it's 'just' 7.

Otiel commented 1 month ago

Hi all,

I'm sorry, but I have bad news.

TL;DR

BandcampDownloader won't work on Windows 7/8/8.1 anymore, you'll have to upgrade to Windows 10/11.

The whole story

I really wanted to fix this because I know a lot of people are still using Windows 7, so I spent quite a bit of time figuring out what was wrong, and why it suddenly stopped working. Good news is I found what's the real issue, bad news is this is not fixable (at least not without going a route I'm not prepared to take).

For those interested in the technical issue, keep reading... This will also serve as memory for my future self.

Certificate Authority

So bandcamp.com made some changes on their servers (around September 12th, according to you guys). Once I found the time to set up a Windows 7 virtual machine, I found out it was a certificate issue (the exception message being The request was aborted: Could not create SSL/TLS secure channel).

I initially thought it was due to a new CA used to issue the new bandcamp.com certificate (naming GlobalSign) that wasn't present on Windows 7, Windows 7 not receiving new updates anymore (support ended on 2020).

The current certificate on bandcamp.com:

image

The Certificate Authority on my Windows 10 machine, which was not present on the Windows 7 VM:

image

So I exported all CA on my Windows 10 using certutil.exe -generateSSTFromWU ./roots.sst, then imported them all in my VM, but the error was still the same. ☹️

TLS 1.2

So back to square one... I then discovered that bandcamp.com doesn't support TLS 1.0 nor TLS 1.1 anymore (good, those protocols have security issues and Microsoft plans to stop support for TLS 1.0 and TLS 1.1 by October 31, 2024), thus only supports TLS 1.2 or 1.3. You can check this with the following command:

openssl s_client -connect google.com:443 -tls1 ⇒ fails
openssl s_client -connect google.com:443 -tls1_1 ⇒ fails
openssl s_client -connect google.com:443 -tls1_2 ⇒ OK
openssl s_client -connect google.com:443 -tls1_3 ⇒ OK

Or using SSL Labs:

image

I also discovered that Windows 7 does not support TLS 1.2 by default and that you have to enable it manually by installing a KB update and modifying the registry. It's not perfectly clear to me if this is still true today, or if some automatic Windows update already enables TLS 1.2, but all those registry keys didn't exist on my Windows 7, so I figured I should create them to be sure.

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.2\Client
Enabled: 1
DisabledByDefault: 0
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework\v4.0.30319
SchUseStrongCrypto: 1
SystemDefaultTlsVersions: 1

I also discovered IIS Crypto which is a great tool to check the protocols and ciphers configured on your machine for SChannel (= the name of the TLS stack on Windows, which is used by BandcampDownloader because I rely on the .NET Framework to perform downloads):

image

After doing all the steps to ensure that TLS 1.2 is correctly enabled on my VM, the error was still the same. ☹️

Cipher suites

Next to check on the list of potential issues for TLS handshake failures are cipher suites.

When a TLS handshake is established, the client (your machine) first announces to the server (bandcamp.com) the TLS version the client supports, along with the cipher suites that can be used. This is sent in what is called a ClientHello message during the TLS exchange. On the other hand, the server also announces the cipher suites it accepts. So I thought "maybe I now have to enable a cipher suite that is not enabled by default on Windows 7".

IIS Crypto also supports displaying the list of cipher suites configured on Windows:

image

Which is nothing more than the values stored in the registry key HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Cryptography\Configuration\Local\SSL\00010002

image

The cipher suites can also be configured using a group policy, but it wasn't active on my VM (thus we can ignore it):

image

And we can check the cipher suites bandcamp.com accepts on SSL Labs:

image

Ignore the cipher suites for TLS 1.3 (which is not supported at all on Windows 7 anyway), the short list for TLS 1.2 is then:

TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256
TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384

When I checked the values of the cipher suites on my Windows 7, it didn't contain any of those three suites. So I modified the registry once again to add those three cipher suites, checked with IIS Crypto, restarted my VM for the 20th time... And after an ultimate test, the error was still the same. ☹️

Network tracing

To this point, I admit that I was quite lost. I was sure I did everything correctly, read lots of articles that indicated the same things to configure, checked and checked again... But I was back to a point where I didn't know exactly what was wrong. Next thing to do was to check the network packets sent / received by BandcampDownloader.

So I added network tracing to the app by adding the configuration to a file named BandcampDownloader.exe.config.

With network tracing, we can extract the ClientHello message sent to the server:

image

With some little edit, we can get the message in this form:

0000 - 16 03 03 00 9A 01 00 00-96 03 03 66 F1 6F E9 8D   ...........f.o..
0010 - 4D DC 9B 27 91 22 1F 59-B0 D3 2B C9 D3 53 06 30   M..'.".Y..+..S.0
0020 - 21 EF 7D 81 82 FA C7 06-20 68 F0 00 00 20 C0 28   !.}..... h... .(
0030 - C0 27 C0 14 C0 13 C0 2C-C0 2B C0 24 C0 23 C0 0A   .'.....,.+.$.#..
0040 - C0 09 00 9D 00 9C 00 3D-00 3C 00 35 00 2F 01 00   .......=.<.5./..
0050 - 00 4D 00 00 00 16 00 14-00 00 11 77 6A 6C 70 2E   .M.........wjlp.
0060 - 62 61 6E 64 63 61 6D 70-2E 63 6F 6D 00 0A 00 08   bandcamp.com....
0070 - 00 06 00 19 00 18 00 17-00 0B 00 02 01 00 00 0D   ................
0080 - 00 14 00 12 06 01 06 03-04 01 05 01 02 01 04 03   ................
0090 - 05 03 02 03 02 02 00 17-00 00 FF 01 00 01 00      ...............

And decode it using a parser like this one:

image

And to my surprise, I saw that the three cipher suites requested by bandcamp.com (remember? TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 - that I configured in Windows 7) are not in the cipherlist sent! 😱 This is the reason it isn't working.

The end

But why those three cipher suites aren't sent even though I've configured them in the registry? Well, simply because they are not supported in Windows 7, as you can see in this Microsoft doc. Same goes for Windows 8 and 8.1. And I found that:

Unfortunately, Windows 7 does not allow adding custom or unsupported cipher suites beyond what is provided by the operating system. The list of supported cipher suites in Windows 7 is fixed, and there is no built-in mechanism to extend or add new cipher suites directly.

Cipher suites are implemented at a deep level in the operating system’s cryptographic stack. Schannel is tightly integrated with the Windows OS and does not provide an extensible mechanism for adding new cipher suites. There is no API or registry setting that allows users to introduce new suites beyond those that Microsoft has already provided.

The cipher suites available are limited to those that Microsoft officially supports, and new suites are usually added only through updates or new Windows versions.

So we're stuck with using some cipher suites that bandcamp.com does not accept anymore to this day. And that is why BandcampDownloader cannot download any page from bandcamp.com anymore.

So here we are, I think I went to the bottom of this debugging. I learned a lot, that took a looooot of time to find the relevant info and official documentation. But I don't think I'll be able to do more, and that means that I have no solution for you guys who are stuck on Windows 7, sorry.

FAQ

alabx commented 1 month ago

It is more complicated than I thought it can be. It is really sad news for me. Anyway I appreciate your hard work and thanks for trying fixing this. I will visit this page sometimes hoping you or someone else find a solution. The one idea I have now is that if bandcamp works via Firefox, Chrome with their TLS, maybe you could just make from you program and an add-on for the browser?

Otiel commented 1 month ago

The one idea I have now is that if bandcamp works via Firefox, Chrome with their TLS, maybe you could just make from you program and an add-on for the browser?

Thanks for the suggestion. However that would require to write a whole new program and I don't have the time (nor the will) to do that, sorry. I suggest you find an alternative if you can't upgrade Windows. There are some online services for instance that provide downloads from Bandcamp.

alabx commented 1 month ago

I understand it, but still I will look here sometimes, maybe you will find a time or other solution will be found. I know these sites, but none can download a full album as easy as yours. BTW I switched from them to your program.

Tindalos commented 1 month ago

That was quite a lot of effort Otiel, and your comprehensive report is appreciated. Perhaps it will be enough for the 7 community to devise a TLS enabling (or TLS proxy) strategy, determine what's successful, and throw a guide together.

We couldn't have started that without knowing what the problem specifically was.

Otiel commented 1 month ago

I'm closing this issue.

Thank you all for reporting the problem and to all of you who provided some info to help.

sdfasfdasdas commented 3 weeks ago

I found a way to make it work. Using Charles Proxy fixes the error. File size appears as 0MB, but the files are downloaded.

2

alabx commented 3 weeks ago

Please show a screenshot with your settings.

sdfasfdasdas commented 2 weeks ago

1 2 You also need to install the certificate from the help menu.

alabx commented 2 weeks ago

I did it like you. It still does not work.

I aslo had a possibility of trying VxKex on Windows 8.1, but the installation does not work. I'm still little too afraid to instal it on my main Windows 7 machine.

Tindalos commented 2 weeks ago

I've been using VxKex on my 7 for ages now, it's very safe.

Was ineffectual for getting BCD to successfully download. Likely because it mostly sorts graphics and .DLL stuff rather than IP stuff.

alabx commented 2 weeks ago

Maybe you have a solution? Can entering a proxy in the program help?

Otiel commented 2 weeks ago

Entering a proxy should fix the issue, but you'll have to find a reliable one (not easy if you want to find a free one) or setup one yourself.