Many issues (sorry)

[abolibibelot1980]

I'm aware that on GitHub each issue is expected to be reported in a separate post, but in this particular case I'll have to make an exception — there are way too many (most of which being general gripes with eMule, only a few being specific to the current “community” 0.60d version), several of them are interconnected, and these days I already have a hard time dealing with my own personal issues, so it's already a significant effort to be able to actually write it down somewhere in a somewhat intelligible fashion. Here is hoping that, for once, I'm doing sumpting that eventually serves a purpose.

– Generally speaking, eMule becomes excruciatingly sluggish when there are many downloads in the queue, starting from approximately 10 000 (and getting seemingly exponentially worse beyond that figure). In particular, it becomes unresponsive for several seconds when a new download is added, or when a download is deleted, or when a download is completed (even a very small file which should be checked in a split second), and it can stay frozen for many minutes when several new downloads are added at once (which interrupts any transfer happening at the same time if more than 2-3 files are added at once), or even when several downloads are resumed. Sometimes after a large number of new downloads have been added, it gets almost stuck, with spikes of high CPU load (13% on my 8 threads CPU which means that the process is mono-threaded, which is certainly sub-optimal) every 5 seconds or so, then it usually becomes somewhat responsive again, but it can last an hour or more, during which the software is virtually unusable. This seems due in large part to a Windows limitation when dealing with folders containing a large number of files (see this for instance), but it is certainly a design flaw which should be remedied in every way possible. [Suggestions] It should allow to to sort temporary files in subfolders, either manually, or automatically based on their assigned category (with the possibility of assigning a category right when downloads are added — with a default suggestion being the name of the search or user tab), or based on the added date (with perhaps a selectable threshold — every month / every week / every day), or create a subfolder for every 1000 downloads (probably the easiest to implement, although not ideal — what happens when a folder gets empty after all downloads in it have been completed?). Also, it might be more efficient not to create 3 new files right away for each new download, but rather store metadata in a single database file, at least until a download is actually started, and only then create actual PART / MET / BAK files (but this might involve a much more extensive rewrite of the current code). As for the BAK files, they're mostly useless, because in pretty much any case of corruption affecting a MET file, the corresponding BAK file is corrupted too (in fact since I set the “secure writing of MET files” option to “always”, whenever I have some corrupted files following a BSOD, those are always BAK files, while MET files are fine — I can see that by examining my backups, corrupted files are totally empty and are compressed to a much smaller size than valid MET / BAK files). [EDIT] Re-reading this a year and a half later: combining my suggestions pertaining to this issue (preserving performance with a large queue) and the one below (naming scheme of temporary files), a better idea would be to: 1) changing the naming scheme from random numbers to ED2K hashes as I already suggested, 2) automatically sorting temporary files in subfolders according to the first character of the ED2K hash, 3) getting rid of the useless BAK files (or at least providing an option to get rid of them). That way, each subfolder would contain about 2/48 of the number of downloads in the queue (2 files per download instead of 3, and 16 subfolders named "0", "1", "2", "3", "4", "5", "6", "7", "8", "9", "A", "B", "C", "D", "E", "F"), so even with a huge queue containing 40,000 downloads (I currently have 32,954) each subfolder would contain about 5,000 temporary files or about 2,500 MET files, which would be much more manageable. (If indeed that performance issue is mostly due to the large number of files in a single folder — oddly, Shareaza doesn't seem to have quite as severe issues with a very large queue, so long as few files have the “pending” status, and even when it does a global pausing of all active downloads because the available space is below the threshold, it takes about half an hour with upwards of 50,000 downloads in the queue, whereas eMule would stay frozen for a whole day with a queue that large, at least as of version 0.60d. And recently I tested version 0.70a: without implementing any such drastic change it vastly improved the processing rate of MET / BAK files, although this processing goes on and on in a seemingly neverending loop while the interface stays frozen, so I had to go back to 0.60d.)

– Having all temporary files in the same folder named with a random number is very awkward anyway. Some months ago, as I had upwards of 40 000 downloads in my queue and it had slowed down to a crawl (most annoyingly, establishing a connection to a server took more than 3 hours — see below), I moved thousands of PART / MET / BAK files into a subfolder, which did the trick, i.e. it was snappy again as it should be; but now if I want to re-add those, there will be thousands of name conflicts. If I reimport them instead, it will be another kind of mess (it would be just as sluggish as adding those downloads anew, and it would change the creation date, thus the added date, to the present date, which I DO NOT WANT). Another issue with this is that, if a new download is added right after a download with the same number is completed (which happens quite often), the creation date (thus the added date) of the new download will be identical to that of the older download, because of a little known Windows quirk called the “creation time tunneling effect” (by design, if a file gets the name of a file which was deleted or moved or renamed less than 15 seconds earlier, its creation date becomes that of the other file). This is definitely unwanted as this messes up the sorting order (normally sorting downloaded files by creation date is the most simple way of sorting them thematically, as files added at the same time were usually found from the same search or the same user list, but files affected by that anomaly will be outliers). [Suggestions] Temporary files should get a unique name based on the file's hash (as Shareaza does) — if this is possible to implement with the current format of MET files (I don't see why not, the actual file name can be much more than the 32 characters that would be required if the PART file name were to be the file's ED2K hash).

– Right after establishing a connection to a server, eMule modifies many MET / BAK files (but not all of them), for no obvious reason, which can last hours if there are many downloads in the queue. I don't know what it is doing exactly, I couldn't find any pattern (it doesn't seem to affect active files only, some which are paused are processed too), and I couldn't find any option to disable this. At some point it took more than 3 hours when I had upwards of 40 000 downloads in the queue, so (as I already mentioned) I had to move those which were inactive into a subfolder (which eMule does not parse), then it was snappy again (adding a few dozens of new downloads took less than a second, as opposed to ~15 seconds per each new download), and the connection was established almost immediately. But then I started to accumulate thousands of new downloads, and it was sluggish all over again. I found a workaround for this at least, albeit a quite clunky one: with a PowerShell script using the command fsutil file createnew, right after I connect to a server, I create a file large enough to fill up the free space minus 1MB (since the free space threshold is set to 1MB), it immediately pauses all active files (which is stupid in and of itself since many of them haven't been “seen complete” even once) and eMule is then ready to work, after I resumed those thousands of files, which conveniently have a specific status “insufficient disk space”. It could be almost satisfying if it didn't add an inordinate amount of clutter to the logs (each file being paused adds a new entry, and again each file being resumed adds a new entry, so each time I do this, at least once a day, it adds several megabytes of that redundant warning, which is becoming an issue in and of itself as I accumulated 330MB of LOG files since the beginning of this year, versus 73MB for the period 2019-2021). [Suggestions] It should not do whatever it is doing that is taking so long.

– As I mentioned above, whenever the free space falls below the threshold, eMule will pause every single active download (or more precisely, every one for which the PART file hasn't reached its expected total size) — this is utterly stupid, at least when it comes to files which haven't been “seen complete” even once. At first this is a quick process, only the current session is affected (if eMule is stopped then, by shutting down the process or because of a f##king BSOD, those downloads will be active at the next session), it doesn't actually modify MET / BAK files, but if free space stays below the threshold long enough (I'm not sure how long exactly, probably an hour — usually when this happened I was either absent or sleeping), then all those downloads will be actually stopped, which means modifying the MET / BAK files, which means, again, having to wait for hours until that s##t is over, or having to shut down the process to end it right away. (That didn't happen since I installed version 0.60d because I'm extra careful not to let free space fall below the threshold — except when I'm provoking this on purpose but then I resume downloads right away —, so I couldn't say if it behaves the same in that regard.) [Suggestions] Only files which have been seen complete at least once should be temporarily paused, and they should not be automatically stopped. [EDIT] In fact, only files which are about to be downloaded should be temporarily paused with an insufficient space warning, there is absolutely no point whatsoever in pausing files which currently have no source.

– A global server search is interrupted very quickly if many files are found (above 100 or so), and/or if files with many sources are found (above 10 or so). Most of the time I want the search to be as thorough as possible, so I have to do multiple searches with multiple size intervals, which is certainly a much larger strain for the servers. [Suggestions] It should allow to continue the search even if many files with many sources are found, or it should have an option to set thresholds higher than the default.

– The StoredSearches.met file is only updated when eMule is closed, so in case of system failure / BSOD (which I get A LOT) all searches made since the last proper shutdown are lost. [Suggestions] The search history should be saved either automatically, with a specified time interval, or after each search (as Shareaza does), or manually at any time.

– The connection to servers as well as Kademlia is utterly unreliable, with no pattern whatsoever. Sometimes the connection to a server is established right away with a “high ID”, sometimes it fails with a dreaded “low ID” (usually when this happens there is a delay of a few seconds before the connection is established), and then it fails repeatedly if I try to disconnect and reconnect to any server for the entirety of the session (It seems correlated with those CPU usage spikes, although it could be totally unrelated). So I have to shut down the emule.exe process (it takes too long to do a proper shutdown, because, again, it takes an inordinate amount of time to do whatever it does with the MET files{1}, I only do it if I want to update the StoredSearches file, which should not be required — see above) and restart eMule and cross fingers, because the next time it can either work right away, or fail again, and I have to start all over again, praying that it will establish a proper connection this time. (And before I found that trick of filling up free space to interrupt that insanely lengthy process, I sometimes waited hours for eMule to be responsive again, only to find the dreaded warning “you have a low ID”.) Same s##t with Kademlia, although both seem totally independant — sometimes both work right away, sometimes connection to server is established normally with a “high ID” but connection to Kademlia fails misebarly, and then it fails again and again for the entirety of the session — and even worse, if it tries and fails long enough, the 4 Kademlia files (key_index.dat, load_index.dat, nodes.dat, src_index.dat) get wiped / reinitialized (stupid really), and so the next time it is guaranteed that it will fail, so I have to restore them from a backup. Sometimes I have to restart that whole process 4 or 5 times (with the simultaneous aggravation of having to 1) wait >2min until that f##king logo gets out of the way; 2) attempt to establish a connection to a server, then if successful create a dummy file with fsutil to fill up the free space on the partition; 3) delete the dummy file (otherwise it will pause those download all over again and add a gazillion entries to the logs) and resume all downloads with the “insufficient space” status; 4) attempt to connect to Kad network) until it finally works. WITH NO EXPLANATION WHATSOEVER. For a long time, I thought that this was due to some kind of corruption (possibly caused by a system failure / BSOD, which I get A LOT as I said{3}) involving one or several of those files, so I would restore those files from a backup any time this happened (I took the habit of doing a backup of the “config” and “logs” directories as well as all the MET and BAK files before each session). But then I realized that interrupting the emule.exe process and restarting eMule could have the same effect... or not! It's totally random, drives me crazy, sometimes it works right away, sometimes it works with a small delay, sometimes it doesn't work at all, and this for a whole session, then I restart and it works fine all over again even though nothing has changed whatsoever in the mean time. A smart man said that expecting a different result from the same experiment in the same conditions was akin to madness (or sumpting like that), well, then using eMule on a daily basis seems to be a recipe towards full-blown madness! è_é {1} For instance on April 6th I had to do a shutdown (yet another SNAFU — the known.met file had been completely wiped as the free space on the system partition had fallen to 0 byte while I was asleep, and it couldn't be recovered automatically once I freed up enough space{2}), and it took 18 minutes until the window was gone (it's much less than the crazy delay after a connection is established, but still way too long). {2} Which begs the question, why isn't there an automatic backup and/or safe write procedure for those crucial known.met / known2_64.met (or StoredSearches, clients, emfriends and other important files), although there is for the downloads.txt file, which is re-created at each startup anyway ? {3} This is not caused by eMule specifically, I've had that kind of issue for years on that damned computer, couldn't find a single serious lead toward a resolution, despite having attempted to request help on at least half a dozen forums. [EDIT] I later found a method that seems to work as reliably as can be: 1) first I temporarily change the minimum free space setting to a value that is above the current value of free space on the partition containing eMule's "temp" and "incoming" folders, which triggers a global pausing (that's more convenient that my earlier method of creating a “dummy” file using a script); 2) I connect to Kademlia (when all downloads are temporarily paused, it usually works right away, sometimes it takes a few seconds, rarely does it fail); 3) I connect to a server (when all active downloads are temporarily paused, it only takes a few seconds, instead of 3+ hours); 4) then I re-change the minimum space setting back to a low value, then I sort downloads by status, select all of those with the “insufficient space” status, and resume them (this has to be done within an hour otherwise it starts the horrendous global pausing process which nothing can stop) — THEN I'm good to go. Another problem is that, even if eMule is set to NOT reconnect automatically, it still does reconnect automatically, sometimes but not always, seemingly at random, when the computer wakes up up from sleep (either regular sleep or deep sleep), it's not consistent, sometimes it does, sometimes it doesn't, I couldn't find any pattern whatsoever, and if that happens it stays frozen, and there is no way to stop that insanely lengthy and useless process, short of shutting down the emule.exe process (even filling up the partition has no effect at that point). I have yet to test if that happens when manually disconnecting eMule before putting the computer to sleep, but even if it does the trick that's yet another thing I'll have to keep in mind, yet another thing that shouldn't have to cause any such trouble.

– That f##king logo appearing right in the middle of the f##king screen at each startup, and staying there for an inordinate amount of time, most likely related with the number of downloads in the queue (I checked a few days ago, it took about 2min30s for that logo to go off), preventing me from doing anything else (at least anything that involves seeing what's underneath) during that precious time (as it stays on the forefront even if the eMule window is put to the background or minimized). [Suggestions] No need for that damn logo, I'm aware that I started eMule — or at least make it possible to get it out of the way. [EDIT] I later found out that clicking on that logo a split second after eMule has started would make it disappear, at least that's a workaround that works, even though it has to be very quick as after that split second the interface gets unresponsive for 2-3 minutes (probably while the download queue is being loaded).

– When scrolling up or down the download queue with the “thumb dragging” method, very often eMule's GUI suddenly becomes unstable with a pop-up warning saying “A required resource was unavailable”. Once that happened, the whole display gets wonky and eMule has to be shut down. This doesn't happen when scrolling with the “clicking the through” method, which is much slower with a large queue. [Suggestions] Fixing that stability issue.

– Since I installed version 0.60d, sometimes the display of the “Searches” tab gets wonky (the outlines of the “Download selection” and “Close all” buttons move all over the place — sometimes hiding items in the list, then disappearing once the underlying item is clicked on — and the last item of the list is hidden), then it stays that way for the entirety of the session. Then it gets back to normal again at the next session. I couldn't find a pattern as to when / why it happens. It never happened with version 0.50a. [Suggestions] That s##t should not happen.

– Sometimes, for no reason (free space is above the threshold), it starts a mindnumbingly long and absurd process of seemingly stopping every single download, at the measly rate of 3 or 4 per minute. This happened once since I installed v. 0.60d, and this time my trick or filling up the partition didn't work (even putting putting the system to sleep didn't stop that process), so I had to wait, and wait, and wait, hoping that it would eventually be over, then more than five hours later, in the middle of the night, I got a fucking BSOD and all the wait was for nothing, and I had to go through a tedious process of restoring the MET / BAK files for which I had a backup, then identifying which ones had no backup and had to be resumed manually. [Suggestions] That s##t should not happen.

– When a download is finished but the file name is too long (exceeds the filesystem's limit), it fails with a misleading status saying that the “disk space” is insufficient, while the progress bar looks the same as when the storage space is actually insufficient (with black and yellow stripes). If I rename the file with a shorter name, then it gets completed right away. [Suggestions] It should accurately report that the name is too long and propose to shorten it.

– Priorities are not always respected. For instance, I got the list of files shared by one user, added many of them, then painstakingly put some in “high” priority, some in “normal”, and some in “low” priority (which is way too slow for reasons already exposed), but then files in “low” priority were downloaded before files in “high” priority. Speaking of which, it doesn't make sense to add all downloads in "high" priority, default should be "normal". Another somewhat related weirdness: if I added a file to the download queue weeks ago, which hasn't started yet although it is active (not paused / stopped), and then I get the list of shared files from a user, which includes that very same file (so that file does have at least one currently connected source), if I add new downloads from that list, they may get started right away, and all of them may get downloaded in a row, but the older file which was added weeks ago still does not start. Then if I delete that file from the queue, and re-add it from the Search tab, it gets downloaded right away. Like Chewbacca, this doesn't make any sense.

– Downloading rate is not always respected if it is set to a very low value. Sometimes I get a file from a search, and then I want to see the list of other shared files from the same source, but if it is a very small file (say less than 50KB) and it gets completed right away I don't have time to do that, so I tried to intentionally lower the downloading rate to 1ko/s, but it doesn't work. [Suggestions] If downloading rate is set to 1kops, it should be effectively no more than 1kops. Or, to solve that particular issue, it should be possible to display a source client's shared files even after the file has been completed.

– When copying the contents of a shared files list to a text file, the corresponding folders do not appear, so for the list to be neatly sorted I have to spend an inordinate amount of time adding each folder name. Then if the list is saved in the "StoredSearches" file (which requires that eMule be shut down as I already mentioned), at the next startup, the folders no longer appear at all and it's just a list of unsorted files, so for instance if there are many music albums with file names starting with the track number, there can be two dozen of "01 - " files, then two dozen of "02 - " files, and so on. [Suggestions] When copying a list, it should copy everything that is visible, or at least include the folder names. And the folder names should be saved and restored as well.

– When files in the download queue are sorted by “last seen complete” date, those which have been seen complete recently are now constantly re-sorted, which messes up the layout when trying to display which client is associated with which file, in order to request their list of shared files. That didn't happen with version 0.50b (if a file was “seen complete”, it moved at the top / bottom of the list and stayed at the same spot, only if it turned to red — no source currently connected — and then back to blue, did it get moved again).

– There should be options to control which parts should get downloaded first, or at least to download specific files in sequential order. This is mainly a problem for “solid” compressed archives: nothing can be extracted beyond the first missing chunk, so it's pointless to download chunks located further, in case the file has no complete source anymore it's a waste of space. Also, sometimes I need to download specific parts, in order to check different versions of the same files and be able to determine which one is correct and which one is corrupted — to do so without needlessly downloading redundant chunks I have to edit the MET file myself (despite the warning not to “mess with MET files”), which then tricks potential clients attempting to download that same file into considering as valid chunks which are actually empty on my side (as an example, there was a 4,3GB MKV file which I wanted to check against a file from the same movie with the exact same size but a different hash, and I knew from comparing hash tables from both versions that they had only 5 mismatched chunks, so I edited the MET in order to consider all other chunks as already downloaded, I've already done that successfully, but in this case the only source for that particular file never came back, and so over a few months I uploaded a whopping 436GB of invalid data). Also there should be an option to disable specific files in the download queue from being shared — not only for that kind of admittedly highly peculiar circumstance, but also because the files I want to share first and foremost are those that are rare, while I may download files which already have enough sources and I don't want to waste my measly bandwidth to share these.

– There should be an optional column to sort downloads by extension / type.

If there is a “mod” that already implements at least some of the changes requested above (the most important would be preserving a decent performance when working with a large queue), please point me to it ASAFP. Thanks.

[abolibibelot1980]

Well, that was useful... é_è Has anybody READ this at least ?

[rd1]

Try the new 0.70a beta just released. maybe it'll fix some of those issues.

[aleste81]

I second all these issues. Big slowdown where working with 10 000 + files.

0.70 crash randomly on my Windows 7

[abolibibelot1980]

I second all these issues. Big slowdown where working with 10 000 + files.

0.70 crash randomly on my Windows 7

Re-reading a year and a half later, I'm glad that at least one person shares the same aggravation and frustrations.

I tried version 0.70a on 2023-12-26, and it was a one night ordeal which unfortunately had lasting consequences. – On the positive side, the processing rate of MET/BAK files has vastly improved, now hundreds of those are processed every minute as opposed to 3-4 pairs every minute, BUT the processing has gone completely crazy, now it seems like MET-BAK files corresponding to every single active download is being CONSTANTLY updated (active = not paused or stopped, but it does affect downloads which are temporarily interrupted with the “insufficient space” status so my usual trick of setting the free space threshold higher than the current free space on the partition does not work), which completely freezes the interface, this sh*t starts right after a connection to a server and seemingly NEVER STOPS. (I checked the "temp" directory while it was happening, noted the modification date of one particular MET file, then checked it again at regular intervals, it seemed to be updated every 5 minutes or so, probably the time it took to process all of 12780 pairs of MET-BAK files that were modified, out of 32890, therefore most likely corresponding to non-paused downloads. After a long while I shut down the process and started all over again, with the exact same result, the same 12780 pairs of files were being constantly processed in a neverending loop. After that I gave up and went back to 0.60d. So much for fixing some issues.)

– Language has been set to english, and now that I'm back to v. 0.60d I can no longer set it back to french, it asks me to download language files, although the language DLLs are still in the "lang" directory. I could be alright with using an english language interface (even though I've been used to the french interface for about 20 years — but I can understand that the french eMule community is no longer active enough to properly translate newer versions being released on GitHub only — I could do it myself if only each new version actually improved my user experience), if only it didn't show a STUPID SMILEY at the end of every “finished downloading” notification at the bottom of the window (that kind of thing aggravates me a lot very quickly, it's one of the reasons why I don't want to upgrade to Windows 10 — BSODs are bad enough, but BSODs with smileys is adding insult to injury and would make me want to MurderDeathKill someone — the french translation thankfully got rid of that crap).

– I'm not sure if that's related, but I don't see how it could not be related as nothing else has changed lately: now with version 0.60d, when I load a user's list of shared files, it loads each folder in a separate tab, and empty folders in empty tabs... how utterly stupid is that? It never happened before I tried version 0.70a (and I couldn't get version 0.70a to work at all so I couldn't check if it behaves the same in that regard). Before I launched eMule 0.60d, after unsuccessfully trying eMule 0.70a, I restored all recently modified files from a backup made just before installing eMule 0.70a, and I installed eMule 0.70a by merely replacing the emule.exe executable, then re-installed eMule 0.60d the same way, I didn't touch any other file, so I don't understand what's happening here.

To end on a positive note, here are some advantages that eMule has over Shareaza, which makes both complimentary in my opinion. – eMule is generally way more efficient when it comes to downloading large files; Shareaza's performance quickly drops when downloading several large files simultaneously. – Before recent improvements by “ansani”, Shareaza's latest officiel version 2.7.10.2 did not create “sparse” files, even if set to do so (at least on Windows 7). (This has been fixed in 2.7.10.4.) – eMule allows to sort downloads very conveniently, and to search a specific downloads, either through a search menu obtained with [Ctrl]+[F] or by simply typing the first few character's of the file's name. In Shareaza, there are no “added on” or “last seen complete” columns, so once the current sorting order has been changed, it's impossible to get back to the former sorting order, or to sort downloads in any other useful way, and it offers no way of quickly locating a specific download. – Shareaza's AppData directory gets very large after a few months of regular activity, in particular the "TigerTree.dat" file can reach a humongous size (mine has recently passed 1GB) and never seems to shrink. Despite that, it does not remember all files which were downloaded then removed. As for eMule, it does remember everything that was ever downloaded, with a "known.dat" file that's much smaller (below 8MB on my system, and I've been using eMule for a far longer time than Shareaza).