[statistics] how many times dotdrop install expands

[roachsinai]

Hi @deadc0de6 for me dotdrop install need 6 seconds, which I feel a delay (CPU: Intel i5-3230M (4) @ 3.200GHz). For sure, the time consumed as the count of files managed by dotdrop. So as almost all my dotfiles is managed by dotdrop, more files more times.

hmanx ~ » /usr/bin/dotdrop --cfg=/home/roach/code/Git/dotfiles/config.yaml files -G 2>/dev/null | cut -f2 -d',' | sed 's/^.*://g' | wc -l
62
hmanx ~ » ./tt.zsh

0 dotfile(s) installed.
execution time was 6 s.
hmanx ~ » cat tt.zsh     
#/usr/bin/zsh

start_time=`date +%s`
eval $(cat /home/roach/.dotdrop.xenv /home/roach/Nutstore_Files/我的坚果云/.env | grep -v "^#") /usr/bin/dotdrop --cfg=/home/roach/code/Git/dotfiles/config.yaml install
end_time=`date +%s`
echo execution time was `expr $end_time - $start_time` s.

What I wanna

My pc is bought at 2014, it is old, and cpu is operation rate is low than recent.

So if people shows there cpu and time consumed, people can have a v.s.

And what's more, is there a way to make a time analyzer?

[deadc0de6]

Yeah, it is indeed quite long. I'll try to profile the execution and see how I can improve the speed.

You can always monitor execution time with time:

time ./dotdrop.sh install -f

Are your dotfiles online? I wanted to test the execution time and the profiling on them.

[roachsinai]

Sorry, it's not online, as have some private info like password...

[deadc0de6]

No problem, I'll start profiling and finding bottlenecks and let you know what comes up.

[deadc0de6]

@roachsinai can you give it a try with the latest commit from master and let me know if your perfs improved?

[deadc0de6]

Also make sure to re-install dependencies with

pip install --user -r requirements.txt

[roachsinai]

As a Manjaro user, I should use AUR (en) - dotdrop-git?

[deadc0de6]

you could but you could also direct use dotdrop from source, see this doc.

$ cd /tmp/
$ git clone https://github.com/deadc0de6/dotdrop.git
$ cd dotdrop/
$ pip install --user -r requirements.txt
$ ./dotdrop.sh --cfg <path-to-your-existing-config-file> files

[roachsinai]

hmanx ~ » ./tt.zsh

0 dotfile(s) installed.
execution time was 5 s.
hmanx ~ » ./tt.zsh

0 dotfile(s) installed.
execution time was 5 s.

It did improved, but not so big. Sometimes got 6 seconds, too.

I installed it to system from AUR, so can't use time ./dotdrop.sh install -f,

and I've tried time eval $(cat /home/roach/.dotdrop.xenv /home/roach/Dropbox/.env | grep -v "^#") /usr/bin/dotdrop --cfg=/home/roach/code/Git/dotfiles/config.yaml install but no time used info.

[deadc0de6]

Perfs improvement should be better, can you make sure you have run pip install --user -r requirements.txt and that the python-magic package is installed through pip

pip freeze | grep -i magic

[roachsinai]

No, I installed it manually just right now, check below:

hmanx ~/./y/dotdrop-git master » pqs ruamel  
local/python-ruamel-yaml 0.16.12-1
    YAML parser/emitter that supports roundtrip preservation of comments, seq/map flow style, and map key order
local/python-ruamel.yaml.clib 0.2.2-1
    C version of reader, parser and emitter for ruamel.yaml derived from libyaml
hmanx ~/./y/dotdrop-git master » pqs docopt 
local/python-docopt 0.6.2-7
    Pythonic argument parser, that will make you smile
hmanx ~/./y/dotdrop-git master » pqs jinja 
local/python-jinja 2.11.2-2
    A simple pythonic template language written in Python
hmanx ~ » pqs magic       
local/python-magic-ahupp 0.4.15-2
    A python wrapper for libmagic

But still need 5 seconds to install.

What's worse, after I install magic something wrong with templating:

hmanx ~/c/G/dotfiles master » dpcp
=> compare d_gadgets.d: diffing with "/home/roach/.vim/pack/vimspector/opt/vimspector/gadgets/linux/.gadgets.d"
=> diff "bash.json":
--- /home/roach/.vim/pack/vimspector/opt/vimspector/gadgets/linux/.gadgets.d/bash.json  2020-07-25 16:16:35.937012529 +0800
+++ /tmp/dotdrop-1i6ekxud/home/roach/.vim/pack/vimspector/opt/vimspector/gadgets/linux/.gadgets.d/bash.json     2020-11-06 18:29:23.652601703 +0800
@@ -4,7 +4,7 @@
       "name": "bashdb",
       "command": [
         "node",
-        "/home/roach/.vscode/extensions/rogalmic.bash-debug-0.3.9/out/bashDebug.js"
+        "{{@@ home_ @@}}/.vscode/extensions/{{@@ env['vsc_bash_debug_ext_name'] @@}}{{@@ env['vsc_bash_debug_ext_version'] @@}}/out/bashDebug.js"
       ]
     }
   }

only this dotfile templating no substitute by env variables.

[roachsinai]

Could you give me the magic project link of pypi.org? Seems what I've installed is a wrong one.

What I installed is: https://pypi.org/project/python-magic/

[deadc0de6]

It is present in the requirements.txt file.

pip install --user python-magic

[deadc0de6]

I fixed the issue with json. I'll continue to profile and see if I find other bottlenecks.

[deadc0de6]

Dotdrop will by default process each file through the template engine. This can be slow if you have many files. You could use the template dotfile entry and potentially the template_dotfile_default config entry to force it not to template dotfiles you know are not templates.

Doing that will drastically improve the execution time. But it all depends on your setup, either you have a small number of templates and then you set template_dotfile_default: false and set each of the dotfile using template to template: true or you keep the default for template_dotfile_default (true) and set the dotfile that are not template to template: false.

Here is the doc.

Let me know if that improves your perfs.

[roachsinai]

First, thanks for fix json issue! Yeah I'll try that entry. Just wonder will multiprocess be an alternative? As a lot of my dotfile use templates. If has difference with old dotfile when install or compare, access standard output through mutex.

[deadc0de6]

@roachsinai I made an attempt at concurrent installation of dotfile in the concurrent branch. Can you give it a try and let me know if that improves?

Use for example --workers=10 on install

$ cd /tmp/
$ git clone https://github.com/deadc0de6/dotdrop.git
$ cd dotdrop
$ git checkout concurrent
$ time ./dotdrop.sh install --workers=10 -f --cfg <path-to-your-config>

[roachsinai]

@deadc0de6 it's awesome! It's down to 3 from the very beginning 6 seconds. I use only --workers=4, as my cpu is 2 cores 4 threads.

[deadc0de6]

Nice! I'll merge that branch into master then. But 3 seconds still sounds to me like an eternity, how many dotfiles do you actually handle?

## this gives you the number of dotfiles
$ ./dotdrop.sh detail 2>/dev/null | grep 'dst:' | wc -l

## this gives you the number of actual files
$ ./dotdrop.sh detail 2>/dev/null | grep 'template:' | wc -l

Have you tried working with the template dotfiles entry or template_dotfile_default?

[roachsinai]

😃, maybe I abused template...

hmanx ~ » dotdrop detail 2>/dev/null | grep 'template:' | wc -l
120
hmanx ~ » dotdrop detail 2>/dev/null | grep 'dst:' | wc -l
58

No, didn't use template entry, as maybe 40 of 58 files involved template...

hmanx ~ » cat /home/roach/Nutstore_Files/.env | grep = | wc -l
78

means I have 78 variables, some are same value but different aims so has different variable name.

[roachsinai]

And I thoght the reason of 3 seconds is I have a vimrc file which use templete include 40 files...

So it can't processed in multiprocess.

[roachsinai]

And I thoght the reason of 3 seconds is I have a vimrc file which use templete include 40 files...

So it can't processed in multiprocess.

Seems it's not true as install time not decrease after I removed the vimrc entry.

For now the waiting time is almost not detectable when you have to confirm if has difference with des and src when install.

Thanks a lot!

By the way, will workers option support compare command?

[deadc0de6]

I have added some timing debug logs at the end of the whole process to see relatively (since you have to enable debug) the duration of the config parsing and command execution.

...
[DEBUG][__main__.main] done executing command "install"
[DEBUG][__main__.main] options loaded in 0.424222469329834
[DEBUG][__main__.main] command executed in 1.728538990020752
...

But my guess is that the templating is taking the most time in your case.

Do you think it's worth looking more in depth for what is slowing down your installation process?

[roachsinai]

Yeah, is there a document to follow?

Well, I mean it great for my using, maybe most people don't abuse template. I just want to check it if possible.

[deadc0de6]

Currently compare doesn't support multiple workers but I'll have a look how it could be implemented and will let you know when I have something concrete.

There is no document but I will keep profiling and will let you know. But TBH I have the feeling I'll end up identifying the templating process (outside of dotdrop scope) to be where dotdrop spends most its time.

[roachsinai]

@deadc0de6 Thanks for that. So should I close this issue or keep it open?

[deadc0de6]

Let's keep it open while I implement the workers on compare.

[roachsinai]

Hi @deadc0de6 got error:

[ERR] "-w --workers" must be used with "-f --force", so this means cocurrent means all installation should be confirmed with y(es)?

[deadc0de6]

Exactly, actually all installation processes are executed in parallel so user interaction is removed. You can of course check the status of your dotfiles with compare or even --dry beforehand if you are unsure.

[deadc0de6]

@roachsinai another way of doing would be to simply ignore any dotfile that needs interaction when using -w --workers

[roachsinai]

Oh yeah, prefer to add --force. Thanks for explanation.

[deadc0de6]

@roachsinai I have implemented -w --workers for install, compare and update in the chmod branch. I also made a little documentation on it here.

If you try it out and find issues let me know but also report if it works as expected and allows you reach better performances!

[deadc0de6]

@roachsinai also it might be worth testing this branch on your setup since I made a few improvements for speed.

[roachsinai]

Hi @deadc0de6 thanks for this! But I had an issue: dotdrop compare install to a temporary director even without -t option. Check below:

hmanx ~ » which dotdrop
dotdrop: aliased to eval $(cat /home/roach/.dotdrop.xenv /home/roach/Nutstore_Files/我的坚果云/.env | grep -v "^#") /usr/bin/dotdrop --cfg=/home/roach/code/Git/dotfiles/config.yaml
hmanx ~ » which dpcp
dpcp: aliased to dotdrop compare
hmanx ~ » dpcp
        -> create directory /tmp/dotdrop-w6caewvi/home/roach
        -> create directory /tmp/dotdrop-w6caewvi/home/roach/.proxychains
        -> create directory /tmp/dotdrop-w6caewvi/home/roach/.ipython/profile_default
        -> create directory /tmp/dotdrop-w6caewvi/home/roach/.jupyter
        -> create directory /tmp/dotdrop-w6caewvi/home/roach/.moc
        -> create directory /tmp/dotdrop-w6caewvi/home/roach/.config/mpv
        -> create directory /tmp/dotdrop-w6caewvi/home/roach/Documents/scripts
        -> create directory /tmp/dotdrop-w6caewvi/home/roach/Documents/scripts/setups
        -> create directory /tmp/dotdrop-w6caewvi/home/roach/Documents/scripts/mot
        -> create directory /tmp/dotdrop-w6caewvi/home/roach/.config/neofetch
        -> create directory /tmp/dotdrop-w6caewvi/home/roach/.rednotebook
        -> create directory /tmp/dotdrop-w6caewvi/home/roach/Documents/configs/konsole
        -> create directory /tmp/dotdrop-w6caewvi/home/roach/.cargo
        -> create directory /tmp/dotdrop-w6caewvi/home/roach/.rednotebook/templates
        -> create directory /tmp/dotdrop-w6caewvi/home/roach/.cache/yay/python-lunarcalendar-git
        -> create directory /tmp/dotdrop-w6caewvi/home/roach/.cache/yay/python-pyephem
        -> create directory /tmp/dotdrop-w6caewvi/home/roach/.config/cheat
        -> create directory /tmp/dotdrop-w6caewvi/home/roach/.vim
        -> create directory /tmp/dotdrop-w6caewvi/home/roach/.cache/yay/python-quickfix
        -> create directory /tmp/dotdrop-w6caewvi/home/roach/.config/fcitx5/conf
        -> create directory /tmp/dotdrop-w6caewvi/home/roach/.vimspector
        -> create directory /tmp/dotdrop-w6caewvi/home/roach/.vim/pack/vimspector/opt/vimspector/gadgets/linux/.gadgets.d
        -> create directory /tmp/dotdrop-w6caewvi/home/roach/.config/fontconfig
        -> create directory /tmp/dotdrop-w6caewvi/home/roach/.config/zathura
        -> create directory /tmp/dotdrop-w6caewvi/home/roach/.config/Code/User
        -> create directory /tmp/dotdrop-w6caewvi/home/roach/.vscode/self_configs
        -> create directory /tmp/dotdrop-w6caewvi/home/roach/.config/redshift
        -> create directory /tmp/dotdrop-w6caewvi/home/roach/.local/share/xfce4/terminal/colorschemes
        -> create directory /tmp/dotdrop-w6caewvi/home/roach/.config/asynctask
        -> create directory /tmp/dotdrop-w6caewvi/home/roach/.zsh-completions
        -> create directory /tmp/dotdrop-w6caewvi/home/roach/.tmux/tmuxline
        -> create directory /tmp/dotdrop-w6caewvi/home/roach/.local/share/konsole
        -> create directory /tmp/dotdrop-w6caewvi/home/roach/.config/xfce4/xfconf/xfce-perchannel-xml
        -> create directory /tmp/dotdrop-w6caewvi/home/roach/.config/keepassxc
        -> create directory /tmp/dotdrop-w6caewvi/home/roach/.local/share/fcitx5/data/quickphrase.d

58 dotfile(s) compared.

[deadc0de6]

Yes this probably happens because those dotfiles are templates and in order to compare a template dotdrop needs first to deploy it to a temporary directory. If those are not template though, this is an issue.

[deadc0de6]

I will however hide those logs to avoid any confusion.

[roachsinai]

Yes this probably happens because those dotfiles are templates and in order to compare a template dotdrop needs first to deploy it to a temporary directory. If those are not template though, this is an issue.

Seems link: link_children also create temporary files.

And -> create directory /tmp/dotdrop-w6caewvi/home/roach/.cargo this one neither a link_children nor template but still created.

[deadc0de6]

Thanks a lot, good catch! It should be fixed now. If it isn't, can you provide me with more info so that I can reproduce the issue?

Besides are your timing/performance better with this version?

[roachsinai]

Hi, @deadc0de6 confirmed temporary file created by compare has template.

And preformance better! dotdrop install know is in 1 seconds, great!

But seems dotdrop compare can't recognize -f or --force option.

hmanx ~ » dpcp -f         
Usage:
  dotdrop install   [-VbtfndDa] [-c <path>] [-p <profile>]
                                [-w <nb>] [<key>...]
  dotdrop import    [-Vbdfm]    [-c <path>] [-p <profile>] [-s <path>]
                                [-l <link>] <path>...
  dotdrop compare   [-LVb]      [-c <path>] [-p <profile>]
                                [-w <nb>] [-C <file>...] [-i <pattern>...]
  dotdrop update    [-VbfdkP]   [-c <path>] [-p <profile>]
                                [-w <nb>] [-i <pattern>...] [<path>...]
  dotdrop remove    [-Vbfdk]    [-c <path>] [-p <profile>] [<path>...]
  dotdrop files     [-VbTG]     [-c <path>] [-p <profile>]
  dotdrop detail    [-Vb]       [-c <path>] [-p <profile>] [<key>...]
  dotdrop profiles  [-VbG]      [-c <path>]
  dotdrop --help
  dotdrop --version

[roachsinai]

Maybe you just forget to add [-f] in compare Usage.

And finished compare in 2 seconds only one worker, in 1 seconds for three workers.

Finished compare in 3 seconds only one worker not use chmod branch.

What will you do to suppress the verbose of compare, add a option?

[deadc0de6]

Since compare does not expect use interaction it won't need -f --force when using -w --workers. That's great that the performances improved. Are the performances reaching your expectations?

[roachsinai]

Yes! New version is beyond my expectations.

[deadc0de6]

Nice, happy it improved! This branch will soon be merged into master and then part of the next release. Thanks for your help and patience!

[deadc0de6]

New version is out: v1.4.0

[roachsinai]

@deadc0de6 Thank you so much for all of this!!!