Open andyundso opened 3 years ago
Thanks for the report, a lot of good points there. I don't see most of these issues on Windows. I mentioned before that MacOS and Linux are still super unstable, so let's fix that. I will prioritize these issues for the v1,3 update (comes out in a few days).
Let's see
I think the lack of file system permissions is indeed what causes some of these issues.
The search scan shouldn't take longer than 2 minutes per 1 TB drive. It probably scans all those ghost drives + it gets stuck because MacOS doesn't let it create the index file in the first place. The index files should be created immediately after the scan started.
I think, by the v1.3 update, I can fix at least the following issues from your list:
I will try to figure out how to request Full DIsk Access
permission to avoid all the related issues.
To implement a progress bar, it would need to know how many paths there are on the drive. I will need to research this, see if it's possible to get this number quickly with some terminal or Node or C++ calls.
df
command)?df
command's output?Ctrl (command) + Shift + I
in the app, do you see any errors in the console tab there?I think the lack of file system permissions is indeed what causes some of these issues.
I gave it full disk permissions in the meantime, maybe this will help.
How many drives do you see in the notification during the scan? 13 or 8 (according to the df command)?
I didn't see any notification about a scan.
Which of all those drive you don't actually need to see? Should it display all the drives listed in the df command's output?
It's a bit tricky under Mac OS, as a lot of those volumes are just shenanigans by Apple. This article explains it quite well. I ran diskutil
on my machine with a USB stick attached and it looks as follow:
Andys-MacBook-Pro:Documents apf$ sudo diskutil list
/dev/disk0 (internal, physical):
#: TYPE NAME SIZE IDENTIFIER
0: GUID_partition_scheme *1.0 TB disk0
1: EFI EFI 314.6 MB disk0s1
2: Apple_APFS Container disk1 1.0 TB disk0s2
/dev/disk1 (synthesized):
#: TYPE NAME SIZE IDENTIFIER
0: APFS Container Scheme - +1.0 TB disk1
Physical Store disk0s2
1: APFS Volume Macintosh HD - Data 299.0 GB disk1s1
2: APFS Volume Preboot 354.9 MB disk1s2
3: APFS Volume Recovery 626.1 MB disk1s3
4: APFS Volume VM 20.5 KB disk1s4
5: APFS Volume Macintosh HD 15.3 GB disk1s5
6: APFS Snapshot com.apple.os.update-... 15.3 GB disk1s5s1
/dev/disk2 (external, physical):
#: TYPE NAME SIZE IDENTIFIER
0: GUID_partition_scheme *8.1 GB disk2
1: Microsoft Basic Data UBUNTU 20_0 8.1 GB disk2s1
I would suggest to ignore /dev/disk0
and /dev/disk1
and only make disk2s1
show up in my example, as /dev/disk2
is also not accessible.
If you open the task monitor, during the search scan, what % of the CPU you can see being used there? Is it ~100%?
It's around 80% at the moment.
https://user-images.githubusercontent.com/7010698/128733384-540f2b12-96a4-4aea-ba5b-ccb30b267ef5.mov
Full process information:
/Applications/Sigma file manager.app/Contents/Frameworks/Sigma file manager Helper (Renderer).app/Contents/MacOS/Sigma file manager Helper (Renderer) --type=renderer --field-trial-handle=1718379636,8624344380463294285,208185530941624348,131072 --enable-features=WebComponentsV0Enabled --disable-features=CertVerifierService,CookiesWithoutSameSiteMustBeSecure,SameSiteByDefaultCookies,SpareRendererForSitePerProcess --lang=en-GB --standard-schemes=app --secure-schemes=app --bypasscsp-schemes --cors-schemes --fetch-schemes=app --service-worker-schemes --streaming-schemes --app-path=/Applications/Sigma file manager.app/Contents/Resources/app.asar --no-sandbox --no-zygote --node-integration-in-worker --num-raster-threads=4 --enable-zero-copy --enable-gpu-memory-buffer-compositor-resources --enable-main-frame-before-activation --renderer-client-id=5 --no-v8-untrusted-code-mitigations --shared-files
What do you mean by "you need to navigate into a directory first before Sigma gets permission to read" - do you mean you have to navigate the directory from within the Sigma app and it prompts you with a sudo dialog?
Home directory
Sadly I cannot record a video, once you have it permissions, you cannot re-trigger the prompt.
If you press
Ctrl (command) + Shift + I
in the app, do you see any errors in the console tab there?
Yes, from index.js, line 11:
Uncaught (in promise) Error: EACCES: permission denied, lstat '/Users/apf/Library/Application Support/Sigma file manager/app storage/bin/youtube-dl/youtube-dl
Do you see all or some of these issues on Windows?
I didn't install the app yet on Windows, as I don't have backups yet of that machine.
To implement a progress bar, it would need to know how many paths there are on the drive.
On Mac OS X and Linux, you could use Inodes for the progress bar. This would give you a rough number of all the files and folders on a device. However, a file can have multiple Inodes on older systems.
@andyundso thanks for the help with debugging.
asking for permissions to access your home directory
Do you think it should instead automatically trigger the access prompt several times in a row for each user directory, instead of asking one at a time when you open them yourself? Not sure if this approach would be better though.
I would suggest to ignore /dev/disk0 and /dev/disk1
Yeah, I think I'll just filter out all the /System/
partitions and keep only the removable devices and virtual drives. The only problem is I'm not sure how to tell which drives are "real" and which ones are not.
For example, the USB drives returns 2 entries.
{
name: '/dev/disk2',
identifier: 'disk2',
type: 'virtual',
fsType: '',
mount: '',
size: 3076521984,
physical: 'HDD',
uuid: '',
label: '',
model: 'SanDisk 3.2Gen1',
serial: '',
removable: true,
protocol: 'USB'
},
{
name: '/dev/disk2s1',
identifier: 'disk2s1',
type: 'part',
fsType: 'MS-DOS FAT32',
mount: '/Volumes/ESD-USB',
size: 3076417126,
physical: 'HDD',
uuid: 'A34D8EAC-3733-39FB-9BEB-A9555D01CA32',
label: 'ESD-USB',
model: '',
serial: '',
removable: true,
protocol: 'USB'
}
And I can't just filter out type: 'virtual'
, so, I'll have to figure out a way to distinguish them. Maybe run some extra commands.
The file mentioned in the error is one of the binaries that I pack into the app:
'/Users/apf/Library/Application Support/Sigma file manager/app storage/bin/youtube-dl/youtube-dl'
The app copies these binaries on launch from the internal archive to the app's "app storage" directory so they can be accessed. Perhaps the copy operation is messing up the file permissions and causing this error.
I also see this error on launch.
I just tried to change both the file's and its folder's permissions after copying it from within the app's console, but it looks like it needs sudo permissions for it to work.
require('child_process').exec("chmod 777 '/Users/apf/Library/Application Support/Sigma file manager/app storage/bin/youtube-dl/youtube-dl'")
I just found out that on MacOS, the function that gets the drive list, might be the reason for the high CPU usage that you are seeing. I noticed that it spawns the diskutil
command processes faster than they are finishing, which results in more and more diskutil
processes being created with every call (every second).
Do you also see multiple diskutil
processes after the launch on your computer? Maybe it's just my virtual machine being too slow for it
Hi @aleksey-hoffman
The only problem is I'm not sure how to tell which drives are "real" and which ones are not.
In those cases, I would filter out any empty fsType
drives that show up, as they don't have a file system and cannot be accessed.
I just tried to change both the file's and its folder's permissions after copying it from within the app's console, but it looks like it needs sudo permissions for it to work.
Maybe it also lacks permissions. I can check this today and get back to you.
Do you also see multiple
diskutil
processes after the launch on your computer? Maybe it's just my virtual machine being too slow for it
No, but I see that it is spawning diskutil after diskutil process.
No, but I see that it is spawning diskutil after diskutil process.
Does the amount of diskutil
processes keep growing as the app stays open or does it stay at 0-1 processes all the time?
On my MacOS VM, in the default task monitor app, I can see the diskutil
process amount keep increasing, which leads to high CPU usage. I thought that might have been the reason you were seeing high CPU usage as well,
It seems, on slow machines, the app spawns diskutil
processes faster than the previous ones are getting closed. Perhaps I should make it check the drive stats every 2 seconds instead. Though, in this case, the user will have to wait for 0 - 2 sec for a connected drive to show up on the list, instead of just 0 - 1 sec.
Can you please check if the default Alt (i.e Option) + [1 - 9]
shortcuts work on MacOS?
Option + 1
When I press Option + 1
on my Mac VM it shows notifications saying that "#!" doesn't exist. I'm not sure if it's a problem with VM's input or if the Option key actually modifies digit input,
The key
property shows as !
and ™
for Option + 1
and Option + 2
respectively, instead of just 1
and 2
:
document.addEventListener('keydown', event => console.log(event))
I should probably rename Alt to Option on Mac as well.
clicking something in the app takes about 2 seconds to load, same for animations.
Just to clarify, do you see this problem only when there's an ongoing search?
It shouldn't lag at all, the search is running on a separate thread. On Windows everything looks fine. It's strange that it's lagging on MacOS. I will have to make sure it's not related to permissions as well. I figured out a way to grant the app full disk access permissions.
Just so you know, I solved the problem with phantom drives and the console error for binaries. This will fix the problem with image preview generation, drag & drop video downloading, and file archiving.
I will finish implementing fixes by tomorrow and publish changes in the v1.3 update on Friday or Saturday along with many other fixes and new features.
@andyundso an off-topic question, what other features would you like to be prioritized? If you could pick any features you want, including things like "bug fixes for platform X"
Hi @aleksey-hoffman
Sorry for the late feedback.
The amount of diskutil
processes is usually 1, sometimes 2. But I have a relatively high performing Mac Book Pro, so this may not be representative for e.g. Mac Book Airs or iMacs.
Implementation wise, spawning processes regularly could lead to a lot of unnecessary load on a system when there aren't any changes. I'm no expert in this area, but usually you want to either monitor /dev
directory for changes (and if there are relevant changes, you could call diskutil
) or listen for system interrupts. node-usb-detection implements such a system for USB-device only where it provides a callback when a new USB device was detected. I'm sure Nautilus has somewhere a similiar system in their source code which would work not only for USB devices but also in cases where new partitions have been created on a device.
Got the same message as you ("#!" doesn't exist). VMs usually map the windows key to command correctly (from my experience using Windows in a VM on Mac).
I don't have a specific feature request at the moment, since the issue with the high CPU load doesn't really make it fun to use the app and therefore I didn't use it that much. However, I would be interested to try out a preview build with the mentioned fixes. Let me know when you pushed the changes so I can build the app on my own to try it out (also to give you feedback regarding the high CPU usage).
@andyundso good day
spawning processes regularly could lead to a lot of unnecessary load on a system
For sure, I agree. Currently I'm using "systeminformation" NPM module as a temporary solution. I am going to optimize it, no doubt.
The app uses "chokidar" watcher module to detect directory changes. I might have to use it to detect drive changes as well.
Got the same message as you ("#!" doesn't exist).
Thanks. I will fix that as well.
Let me know when you pushed the changes so I can build the app on my own to try it out
Thanks, will do.
Does it lag and have high CPU usage only when you're using search, or all the time? Does it start happening right after launch?
It's unfortunate that Mac users currently have to experience all these early development problems.
Hopefully you will have a much better experience with the Windows build. It shouldn't have performance issues, the problem with shortcuts and the drives.
Does it lag and have high CPU usage only when you're using search, or all the time? Does it start happening right after launch?
Right after launch.
It's unfortunate that Mac users currently have to experience all these early development problems.
Don't worry about it, as you mentioned on the README you mainly developed on Windows and it's fine that some things do not work yet on Mac.
@andyundso ok, if it's lagging all the time then it's most likely caused either by hardware acceleration or diskutil.
I just disabled both of them temporarily and pushed changes to git. Could you please check the fixes one at a time on your real MacOS machine and see if it helps:
Here's the instructions on how to get the repo, run it on the dev server without building the production build, and apply the fixes one at a time,
Please let me know If you unable to run the dev builds, in this case I will build it on my machine and send you the builds.
Please switch Node.js version to 15.14.0
or lower
git clone https://github.com/aleksey-hoffman/sigma-file-manager.git
cd ./sigma-file-manager
npm install
npm run electron:serve
Close the dev server and apply the potential fix 1 (It disables diskutil
):
git checkout ba8768f
npm run electron:serve
diskutil
and hardware acceleration
):git checkout bf5d2ff
npm run electron:serve
@andyundso good day, have you had a chance to try the fixes yet?
Hi @aleksey-hoffman
I did some testing, sorry for the late response.
First of all, Node 15 didn't work for me.
ERROR Failed to compile with 1 error 5:17:39 PM
This dependency was not found:
* fs/promises in ./node_modules/trammel/lib/trammel.js
To install it, you can run: npm install --save fs/promises
However, Node 14 worked. If you don't mind, I would create a PR with a .nvmrc
pointing to Node 14.
Since I cloned the latest main
branch, both hardware acceleration and disk util were disabled. When running the development server, the experience was a bit better than before, but still laggy. However, the CPU usage was mostly normal.
I made a build of the Electron app, installed it and tried it out. The CPU usage is the same as with the current release build.
To have a comparison, I installed the app on Windows and on my Ubuntu machine. For both OS, it worked fine (no lags, no extraordinary CPU usage, Linux actually even better than Windows), so I think it is a specific Electron / Mac OS thing.
I browsed the web for a bit, those are resources I found:
Are you able to reproduce this CPU issue on your Mac VM? If not, I could invest some time into checking what animation could be blocking in Mac OS with the Vue DevTools, but I'm away the next 2.5 weeks for exams at university. However, I think you shouldn't delay your 1.3 release, but write in the release notes that the Mac build has issues with CPU usage.
@andyundso thank you for the investigation.
Are you able to reproduce this CPU issue on your Mac VM?
On my Big Sur VM. performance improves a lot when I apply both fixes.
But I think disabling hardware acceleration only helps on VMs because they only have 128 MB of RAM. I think when it comes to real machines, disabling hardware acceleration might actually decrease performance. I think on your machine what happend was fix 1
improved the performance and then fix 2
made it worse, resulting in a slightly better performance overall.
I also discovered that on MacOS performance drops when you're navigating directories with images, because it's trying to generate thumbs and failing. The MacOS build currently doesn't come with ffmpeg binary (I couldn't find one that would work without triggering MacOS security warning).
Since a lot of people are seeing this issue in other apps, it might actually be caused by Chromium. we should try to upgrade Electron to v14.0.0-beta.21
and see if that helps.
However, Node 14 worked. If you don't mind, I would create a PR with a .nvmrc pointing to Node 14.
Sure, go ahead
I'm away the next 2.5 weeks for exams at university.
Good luck on the exams, you should prepare, so don't worry about the app for now, we can continue investigation after the exams are over.
Hi @aleksey-hoffman
Sorry for the outstanding feedback on my end. I pulled the latest master today to see where things are with Mac OS X and the performance isn't really better. I saw you mentioned on other issues that you'll work on upgrading Electron.
I don't know if you want to keep this issue open until the performance issue is fixed or if we should close it and open a more specific one after the Electron upgrade in case performance still isn't better. It's up to you.
On another note, I don't see anything in any directory, looks like something with the DirItem
component is broken. Are you aware of this?
@andyundso hey mate, good to hear from you! Did you pass the exams?
Sorry, I noticed you left Patreon, I'm sorry if I disappointed you with this project, mate. Thank you for supporting my work in the previous month! I didn't have much time in the past few weeks, but I will try to fix this performance issue for the next update.
The latest master uses old Electron version. Could you try updating it on your Mac?
For this, you would need to run npm install electron@^15.3.0
(make sure the version gets updated in package.json
after this command is done)
After you update the version, make sure you close all the Electron
and Sigma file manager
processes from the task monitor and then run the app again with: npm run electron:serve
I think I might need your help with this performance issue though. Could you maybe record a short video using OBS or something like that to demonstrate the problem? We could also do a Zoom call with screen sharing.
Thanks for reporting the issue with DirItem.vue
, I just fixed it on the main branch
Though there's still an issue with directory list scrolling. I have a lot of changes locally. Once I push everything tp github, this will get fixed as well
@andyundso hey mate, good to hear from you! Did you pass the exams?
I did, thanks for asking.
Sorry, I noticed you left Patreon, I'm sorry if I disappointed you with this project, mate. I didn't have much time in the past few weeks, but I will try to fix this performance issue for the next update.
Oh, I didn't intentionally leave the Patreon. I recently changed my credit card and forgot to update it on Patreon, however, I never got a notification that billing didn't work. Will re-join later this evening. I actually use the app on Windows, I have a few things there, will post the issues sometime this weekend.
The latest master uses old Electron version. Could you try updating it on your Mac? For this, you would need to run
npm install electron@^15.3.0
(make sure the version gets updated inpackage.json
after this command is done)After you update the version, make sure you close all the
Electron
andSigma file manager
processes from the task monitor and then run the app again with:npm run electron:serve
I did update Electron to v15, however, it doesn't solve the performance issue.
Performance on current master with v12:
https://user-images.githubusercontent.com/7010698/139472399-d99753c6-05f9-4d5b-9301-c4a135625640.mov
Performance on current master v15:
https://user-images.githubusercontent.com/7010698/139472426-4b1a4e4f-3fff-49ea-bb9e-a8be3531b3b3.mov
I tried to install the VueDevTools within Electron to see if there's maybe a specific component that causes the issue. However, electron-devtools-installer is broken below Electron v16. If I upgrade Electron to v16, I run into issues with the remote module.
I saw you noted a Todo in the code that you want to move away from @electron/remote
. Is that something you'll do shortly anyways? Otherwise, I would fork the devtools-installer and implement the fix that is mentioned in the issue so I can continue with debugging the performance troubles.
Oh, I didn't intentionally leave the Patreon
Ah, ok. That's good to hear mate, I thought maybe I did something wrong and you got disappointed in the app. Good to hear that's not the case. This gives me motivation to keep improving the app :)
Performance on current master v15
This is really weird. I have better performance on a virtual MacOS with just 128 mb of video memory:
move away from @electron/remote
Yeah, this is problematic at the moment. I'm not sure why @electron/remote
module is not working in Electron 16+.
I just checked. In order to migrate from @electron/remote
I would have to refactor 8 functions I'm not sure how long that will take, I think it might take about 8 hours.
I tried to install the VueDevTools
I think we can try to debug it using the regular DevTools. The app has good performance in a virtual machine, so it seems the problem is caused by the specific configuration of you machine. It might be a problem with accessing GPU, or perhaps there's some infinite loop in the code somewhere. I think we just need to go through functions in App.vue > mounted ()
and disable them one by one to determine which function causes this performance issue.
Can you please add this line to /electronMain.js
and save the file. It should automatically relaunch the app and use the high performance GPU on your machine:
electron.app.commandLine.appendSwitch('force_high_performance_gpu')
@andyundso do you think we could connect on Zoom or some other similar app and take a look at the code together on your machine?
sure, I'm quite busy next week but we can schedule something for the week starting on the 8th of November. What timezone are you living in ?
I'm usually online 8:00 - 22:00 CET (or 7:00 - 21:00 UTC)
@andyundso hey man, let me know when you can jump on a call so we can take a look at the performance issue
@aleksey-hoffman oh my bad. I would have capacity tomorrow after 5pm CET if that works for you?
@andyundso no worries, I've also been quite busy for the past few weeks, implementing new features :) Perfect, 5pm CET tomorrow works for me as well. Do you have Zoom or Skype installed or something like that? We could use Discord as well, but I don't think it allows you to give the mouse and keyboard control
@andyundso btw, I found that Electron apps can become slow on MacOS in these cases:
Could you please test it out when you have a minute? Maybe try something like this:
- If you have an external monitor attached
- If you're using a custom color profile for your monitor.
Interesting stuff 😅 I'll try it out later today.
Do you have Zoom or Skype installed or something like that?
I have a Zoom license. Shall I sent an invite for tomorrow?
Interesting stuff 😅 I'll try it out later today
Yeah, I'll be surprised if that's actually what's happening.
I just sent you a friend request on Discord. I think it will be better to send the link there
So I tried it without an external minor, color profile is the one set by Mac OS X, same effect.
Ok, let's try debugging it then. Are we still planning for 5pm today?
Is there an existing issue for this?
Problem description
Hi, I have some initial feedback from using the app on OS X. These are probably multiple issues, but I wasn't really sure what the underlying problems are, so I didn't split it myself.
Security settings
OS X blocks the app per default from reading anything. So you need to navigate once into your home directory so you can permit Sigma to read your home directory, same for removable devices and network storages. Maybe there is a way that you can check those consents on startup? There is also a permission available called
Full Disk Access
, which would probably solve this issue as well."Phantom" drives
Sigma currently detects a bunch of volumes:
Don't know where this information is coming from. Running
df -h
returns less volumes:I guess querying
/
and ignoring all other disk1's should be the default setting. I also have FileVault active, in case that's relevant.Search
There are a few things related to the search.
Steps to reproduce
Described above.
Environment