excalibur1234 / pacui

Bash script providing advanced Pacman and Yay/Pikaur/Aurman/Pakku/Trizen/Pacaur/Pamac-cli functionality in a simple UI
GNU General Public License v3.0
169 stars 13 forks source link

func_b improvement to use variables #14

Closed thefallenrat closed 7 years ago

thefallenrat commented 7 years ago

Hey, I just made a pull request that modifies func_b to use variables (mostly sadly) to resolve #12

But sadly I got a dead-end in the while loop. In the while loop, I can't make it to store all output into the variable, only the latest output that made it to the variable. So I decide to keep the temporary files until I figured out how to store variables in while loop

P.S Can I add my name in it?

excalibur1234 commented 7 years ago

i have also worked on func_b recently and added support for pacui's default cache directory for AUR files. the core logic seems to work, but there is something wrong with the output format (using files or variables, lists separated by newlines or spaces) and i am not able to install/remove packages. i still have to investigate this little problem.

i will probably be really busy with real life for the next 2 weeks. when i have some time, i will take a look at this PR and merge your code manually with mine. but when you are in a hurry, you can also manually merge my changes into your code. here is my current func_b, just in case:

function func_b
{
        # declare local variables
        local {cache,cachePACAUR,pkgR,pkgI,pkgD,line,temp1,temp2,temp3}

        # get pacman's cache directory from file /etc/pacman.conf (without any white spaces) and write results to variable "cache":
        cache=$( awk -F '=' '/^CacheDir/ {gsub(" ","",$2); print $2}' '/etc/pacman.conf' )
        # set "cache" varialbe to default directory, if  variable "cache" is empty:
        if [[ -z $cache ]]      # if "cache" variable is empty
        then
            cache="/var/cache/pacman/pkg/"
        fi

        if [[ -e /usr/bin/pacaur ]]     # checks, whether file "pacaur" exists, i.e. pacaur is installed
        then 
            # the cache location of pacaur is important for downgrading packages installed from the AUR:
            if [[ -z $AURDEST ]]    # $AURDEST is environment variable for changing pacaur's default cache directory. check, if "AURDEST" variable is empty
            then
                cachePACAUR="$HOME/.cache/pacaur/"
            else
                cachePACAUR="$AURDEST"
            fi
        fi

        # 1. get list of last installs/upgrades/removes from pacman log and display result in fzf. in the end, fzf saves all selected package names (including the words "installed/upgraded/removed/downgraded" to file /tmp/pacui-cache-packages
        tail -3000 /var/log/pacman.log | grep "\[ALPM\]" | grep "installed\|removed\|upgraded" | sed 's/\[ALPM\] //g' | fzf-tmux -m -e +s -i -1 --query="$input2" --cycle --tac --reverse --margin=4%,1%,1%,2% --inline-info --header="TAB key to (un)select. ENTER key to roll back. ESC or CTRL+C to quit." --prompt='Enter string to filter displayed list of recent Pacman changes > ' | sed 's/ ([^)]*)//g' | awk '{print $3 " " $4}' > /tmp/pacui-cache-packages

        # only run the command inside the if-statement, if file /tmp/pacui-cache-packages is not empty and exists - this happens when fzf is quit with ESC or CTRL+C
        if [[ -s /tmp/pacui-cache-packages ]]
        then

            # 2. in case of conflicting packages, packages have to be first removed (with the force option, because other packages might still depend on them).
            #filter /tmp/pacui-cache-packages file for the word "installed" and write package names to variable "pkg"
            pkgR=$( awk '/installed/ {print $2}' '/tmp/pacui-cache-packages' )

            if [[ -n $pkgR ]]     # this if-condition avoids error message when no package gets removed (and $pkgR is empty)
            then
                sudo pacman -Rdd "$pkgR" --color always --noconfirm
            fi

            # 3. in case an "upgraded" package needs a package as dependency, the "removed" packages have to be installed.
            #filter /tmp/pacui-cache-packages file for the word "removed" and write package names to file /tmp/pacui-cache-install
            awk '/removed/ {print $2}' '/tmp/pacui-cache-packages' > /tmp/pacui-cache-install

            if [[ -s /tmp/pacui-cache-install ]]     # this if-condition avoids error messages when no package gets installed (and /tmp/pacui-cache-install is empty)
            then

                # the file /tmp/pacui-install will later contain a list of file names of (old versions of) packages in cache. it has to be empty in the beginning!
                [[ -e /tmp/pacui-install ]] && rm /tmp/pacui-install           # check, whether file exists and delete it.

                # read line by line from file /tmp/pacui-cache-install in while loop and save that line to variable $line
                while IFS='' read -r line || [[ -n "$line" ]]
                do

                    # write name of latest version in pacman cache into file /tmp/pacui-install ("sort" puts latest version on top, which is then selected): 
                    find "$cache" -name "${line}-[0-9]*[0-9a-z.-_]*.pkg.tar.[gx]z" | sort -r | sed -n '1p' >> /tmp/pacui-install

                    if [[ -e /usr/bin/pacaur ]]     # checks, whether file "pacaur" exists, i.e. pacaur is installed
                    then 
                        # do the same as above for files from pacaur's cache directory.
                        # the problem here is that AUR packages are not named/numbered in a constant and easy sortable way. therefore, we search for all files and output their modification date in an easy searchable format (and then, the file name).
                        # then, "grep" is used to get only package files. then, the list is sorted (by the modification date).
                        # awk gets rid of the modification date. grep filters for the file name $line. sed only chooses the first/top line.
                        find "$cachePACAUR" -maxdepth 2 -mindepth 2 -type f -printf "%T+\t%p\n" | grep ".pkg.tar.[gx]z$" | sort -rn | awk '{print $2}' | grep "$line""-" | sed -n '1p' >> /tmp/pacui-temp-downgrade
                    fi

                done < "/tmp/pacui-cache-install"

                # write file into variable "pkgI" (otherwise, the "sudo pacman -U" command does not work) and unique sort /tmp/pacui-install (=remove duplicate lines without ordering the output) 
                pkgI=$( sort -u /tmp/pacui-install )

                # install cannot be done as dependency, because sometimes packages are simply replaced by other packages. in this case, installing as dependency would be bad!
                sudo pacman -U "$pkgI" --color always --noconfirm

            fi

            # 4. filter /tmp/pacui-cache-packages file for the word "upgraded" and write package names to file /tmp/pacui-cache-downgrade
            # /tmp/pacui-cache-downgrade contains list of package names to be downgraded!
            awk '/upgraded/ {print $2}' '/tmp/pacui-cache-packages' > /tmp/pacui-cache-downgrade

            if [[ -s /tmp/pacui-cache-downgrade ]]     # this if-condition avoids error messages when no package gets downgraded (and /tmp/pacui-cache-downgrade is empty)
            then

                # first, count the number of times the package name appears in file /tmp/pacui-cache-downgrade:
                sort /tmp/pacui-cache-downgrade | uniq -c > /tmp/pacui-cache-downgrade-counted
                # first argument in /tmp/pacui-cache-downgrade-counted is the number of times the package name appears and the second is the package name.

                # the file /tmp/pacui-downgrade will later contain a list of file names of (old versions of) packages in cache. it has to be empty in the beginning!
                [[ -e /tmp/pacui-downgrade ]] && rm /tmp/pacui-downgrade           # check, whether file exists and delete it.

                # read line by line from file /tmp/pacui-cache-downgrade-counted in while loop and save that line to variable $line
                while read -r line && [[ -n "$line" ]]
                do

                    # attention, the following variables can be empty:
                    temp1=$( echo "$line" | awk '{print $1}' )   # this variable is the no. of times a package has to be downgraded
                    temp2=$( echo "$line" | awk '{print $2}' )   # this variable is the package name to be downgraded

                    if [[ -n $temp2 ]]   # checks, if variable is not empty
                    then
                        # write list with all versions of package in cache into file /tmp/pacui-temp-downgrade (sorted - newest package version is on top).
                        # this does not work for AUR packages installed by pacaur!
                        find "$cache" -name "${temp2}-[0-9]*[0-9a-z.-_]*.pkg.tar.[gx]z" | sort -r >> /tmp/pacui-temp-downgrade

                        if [[ -e /usr/bin/pacaur ]]     # checks, whether file "pacaur" exists, i.e. pacaur is installed
                        then 
                            # do the same as above for files from pacaur's cache directory.
                            # the problem here is that AUR packages are not named/numbered in a constant and easy sortable way. therefore, we search for all files and output their modification date in an easy searchable format (and then, the file name).
                            # then, "grep" is used to get only package files. then, the list is sorted (by the modification date).
                            # awk gets rid of the modification date. grep filters for the file name $temp2.
                            find "$cachePACAUR" -maxdepth 2 -mindepth 2 -type f -printf "%T+\t%p\n" | grep ".pkg.tar.[gx]z$" | sort -rn | awk '{print $2}' | grep "$temp2""-" >> /tmp/pacui-temp-downgrade
                        fi

                        # temp3 is supposed to be "2p" when temp1=1 and "3p" when temp1=2 ...  --> needed for "sed" command below
                        temp3="$(( temp1 + 1 ))p"
                        # the next line moves the $((temp3-1))-th version below the currently installed package version to file /tmp/pacui-downgrade. if no such old version is available, nothing happens.
                        # this command determines the currently installed verions of package $temp2:  pacman -Q "$temp2" | awk '{print $2}'
                        grep "$( pacman -Q "$temp2" | awk '{print $2}' )" -A 100 "/tmp/pacui-temp-downgrade" | sed -n "$temp3" >> /tmp/pacui-downgrade
                    fi

                done < "/tmp/pacui-cache-downgrade-counted" 

                # write file into variable (otherwise, the "sudo pacman -U" command does not work)
                pkgD=$( cat /tmp/pacui-downgrade )

                # the following if-statement prevents the following error, in case there is no older package version available: "error: no targets specified (use -h for help)"
                if [[ -n $pkgD ]]   # checks, if variable is not empty
                then
                    # downgrade packages by manually installing them: (sudo pacman -U --noconfirm --color always )
                    sudo pacman -U "$pkgD" --color always --noconfirm
                fi

            fi

        fi
}
thefallenrat commented 7 years ago

Alright the code are worked 100% even though there are still some temp files that are not yet replaced by variables ( Caused they're in the while and for loop) . If you want to test it just simply copy the func_b into a new text, then replaces pacman commands :

sudo pacman -Rdd "$pkgR" --color always --noconfirm

into

echo "sudo pacman -Rdd $pkgR --color always --noconfirm"

Run it and then you willl see something like this :

~ >>> ./test                                                                                                                                              
sudo pacman -Rdd aurvote cmake gedit gedit-plugins gspell jsoncpp pcmanfm sakura-gtk2 vim vte xfdesktop yaourt youtube-dl --color always --noconfirm
sudo pacman -U /home/thefallenrat/.pacmancache/gvim-8.0.0851-1-x86_64.pkg.tar.xz /home/thefallenrat/.pacmancache/notepadqq-1.0.1-1-x86_64.pkg.tar.xz /home/thefallenrat/.pacmancache/sakura-gtk2-2.4.2-3-x86_64.pkg.tar.xz /home/thefallenrat/.pacmancache/vim-omnicppcomplete-0.4.1-8-any.pkg.tar.xz /home/thefallenrat/.pacmancache/xed-1.4.6-1-x86_64.pkg.tar.xz --color always --noconfirm
sudo pacman -U /home/thefallenrat/.pacmancache/gdk-pixbuf2-2.36.8-1-x86_64.pkg.tar.xz /home/thefallenrat/.pacmancache/gtk3-3.22.18-1-x86_64.pkg.tar.xz /home/thefallenrat/.pacmancache/gtk-update-icon-cache-3.22.18-1-x86_64.pkg.tar.xz /home/thefallenrat/.pacmancache/linux412-4.12.6-1-x86_64.pkg.tar.xz /home/thefallenrat/.pacmancache/package-query-1.8-1-x86_64.pkg.tar.xz /home/thefallenrat/.pacmancache/wayland-1.13.0-1-x86_64.pkg.tar.xz --color always --noconfirm
thefallenrat commented 7 years ago

Oops accidentally deleted branch and closed the pull request. Sorry about that! Reopening it ...

Also I just wanna say that, func_b improvement has been 100% completed with full use of variables and mktemp command for the pacui-tmp-downgrade that can't be replaced with variable. The code works as expected on my end and you can merge this anytime you want!

excalibur1234 commented 7 years ago

i have read your changes and there are some things i have noticed:

your other code looks fine.

thefallenrat commented 7 years ago

Thanks for the reply!

is the syntax of trap "rm -rf $pacui_tmp_downgrade" EXIT right? i have not worked with the trap command before, but this looks to me as you want to remove the output of pacui_tmp_downgrade file and not the file itself. am i wrong about this?

Actually trap is a conditional command (like if) that will automatically run the code inside " " if the process get closed abruptly via terminal close or CTRL+C. In this case, when it does gets closed abruptly, it will automatically delete ${pacui_tmp_downgrade} so there won't be file leftover...

https://stackoverflow.com/questions/687014/removing-created-temp-files-in-unexpected-bash-exit

you have commented out the following command (line 605) #find "$cachePACAUR" -maxdepth 2 -mindepth 2 -type f -printf "%T+\t%p\n" | grep ".pkg.tar.[gx]z$" | sort -rn | awk '{print $2}' | grep "$line""-" | sed -n '1p' >> /tmp/pacui-temp-downgrade (please adjust this command to use variables instead of files, before you start to use it). i would actually like to include AUR packages in this section as well. this would enable the user to "roll back" an AUR package, which he has removed. what do you think about that?

If I enabled it (with variable), it will messed up the first while loop causing only the first line that got out from the loop. But don't worry, I will think out another alternative solution for this...

you use a couple of times a strange syntax (example from line 648):find $cache -name "${temp2}-[0-9]*[0-9a-z.-_]*.pkg.tar.[gx]z" | sort -r > ${pacui_tmp_downgrade}. i think this looks ugly and i would rather prefer this code structure: pacui_tmp_downgrade=$(find $cache -name "${temp2}-[0-9]*[0-9a-z.-_]*.pkg.tar.[gx]z" | sort -r ) is there a reason why you chose to use that syntax?

This is important, as it will redirect the output into the file that are set by $pacui_tmp_downgrade}.

So doing this :

> ${pacui_tmp_downgrade}

Is the same as :

> /tmp/pacui-tmp-downgrade.XXXXXXXX
# Where the repeated X s are random letters or numbers that are set by mktemp command

I just tested with the $(). It'll make the loop to "run the directory" :

~/RAT's workspaces/pacui >>> ./pacui.bak
aurvote cmake docbook-xml docbook-xsl gedit gedit-code-assistance gedit-plugins git glib2-docs gnome-code-assistance gspell gtk-doc jsoncpp libieee1284 sakura-gtk2 sane sane-openrc source-highlight vim vte xfce4-dev-tools xfdashboard xfdesktop yaourt youtube-dl

/home/thefallenrat/.pacmancache/gvim-8.0.0851-1-x86_64.pkg.tar.xz /home/thefallenrat/.pacmancache/notepadqq-1.0.1-1-x86_64.pkg.tar.xz /home/thefallenrat/.pacmancache/pcmanfm-1.2.5-2-x86_64.pkg.tar.xz /home/thefallenrat/.pacmancache/ratpoison-1.4.9-1-x86_64.pkg.tar.xz /home/thefallenrat/.pacmancache/sakura-gtk2-2.4.2-3-x86_64.pkg.tar.xz /home/thefallenrat/.pacmancache/vim-omnicppcomplete-0.4.1-8-any.pkg.tar.xz /home/thefallenrat/.pacmancache/xed-1.4.6-1-x86_64.pkg.tar.xz
grep: /home/thefallenrat/.pacmancache/artools-base-0.5-2-any.pkg.tar.xz
/home/thefallenrat/.pacmancache/artools-base-0.5.2-1-any.pkg.tar.xz
/home/thefallenrat/.pacmancache/artools-base-0.5-1-any.pkg.tar.xz
/home/thefallenrat/.pacmancache/artools-base-0.5.1-1-any.pkg.tar.xz
/home/thefallenrat/.pacmancache/artools-base-0.4-1-any.pkg.tar.xz
/home/thefallenrat/.pacmancache/artools-base-0.4.1-1-any.pkg.tar.xz: No such file or directory

So doing the strange syntax > ${pacui_tmp_downgrade} is important.


You can try out my code by just copy func_b entirely and replace every sudo pacman command with :

echo " "
echo "$pkgI" | paste -sd " "
excalibur1234 commented 7 years ago

i have just merged your PR. thanks a lot for your work!

one thing i noticed when i added some comments: you use rm ${pacui_tmp_downgrade} i was always under the impression that you need root privileges to remove something from the /tmp/ directory. have you actually checked this or simply forgotten?

and one more style question regarding parenthesis: you use often ${variable_name} to get the output of a variable (sometimes it is quoted to prevent bugs from variable names containing non-standard characters). is there any difference to $(variable_name) or simply $variable_name?

thefallenrat commented 7 years ago

Thanks for merging it! Appreciated it as always! :3

one thing i noticed when i added some comments: you use rm ${pacui_tmp_downgrade} i was always under the impression that you need root privileges to remove something from the /tmp/ directory. have you actually checked this or simply forgotten?

Yeah actually checked this. You don't need to have a privilege to be able to delete all files that resides in /tmp/ ... Take a look at this example :

~ >>> pacui_tmp_downgrade=$(mktemp /tmp/pacui_tmp_downgrade.XXXXXXXXXX)                                                                                               
~ >>> ls /tmp/                                                                                                                                                        
pacui_tmp_downgrade.rehT6ruYm0  pamac-checkdbs-thefallenrat  ssh-C3Qw3R5Xw4ua
~ >>> rm ${pacui_tmp_downgrade}                                                                                                                                       
~ >>> ls /tmp                                                                                                                                                         
pamac-checkdbs-thefallenrat  ssh-C3Qw3R5Xw4ua

and one more style question regarding parenthesis: you use often ${variable_name} to get the output of a variable (sometimes it is quoted to prevent bugs from variable names containing non-standard characters). is there any difference to $(variable_name) or simply $variable_name?

Actually there is according to this, and I use it so that I can add some letter or in this case /n after it :

~ >>> pacui_test=test  
~ >>> echo "$(pacui_test)\nwhat"                                                                                                                                      
zsh: command not found: pacui_test

what
~ >>> echo "${pacui_test}\nwhat"                                                                                                                                      
test
what

Also I noticed that there are still some /tmp/file that are not yet replaced by variables ( or mktemp ) in your other func (such as func_r ). Can I help you with that?

excalibur1234 commented 7 years ago

thx for your explanations. i just fixed the roll-back option (you were missing a >>) and extended it.

but i am still not able to roll back multiple package updates. you can take a look at this again (it is already late here)? it looks like $pacui_downgrade is fine and contains the list of paths to the files i want to downgrade (separated by a newline). but somehow that list is not converted properly into a space separated list, which pacman can interpret. therefore, pacman gives me the following error (when i try to roll-back the update of "kwrite" and "ktexteditor"):

loading packages...
error: '/var/cache/pacman/pkg/kdoctools-5.36.0-1-x86_64.pkg.tar.xz /var/cache/pacman/pkg/ktexteditor-5.36.0-2-x86_64.pkg.tar.xz': could not find or read package
excalibur1234 commented 7 years ago

Also I noticed that there are still some /tmp/file that are not yet replaced by variables ( or mktemp ) in your other func (such as func_r ). Can I help you with that?

are you talking about this file: /tmp/pacui-packages-group

as far as i can see it, it is not security critical, because it is piped into fzf and then stored in a variable. only this variable is fed into pacman with sudo privileges.

am i wrong?

excalibur1234 commented 7 years ago

i have just updated my previous post with an error description. i hope this helps. good night.

thefallenrat commented 7 years ago

it looks like $pacui_downgrade is fine and contains the list of paths to the files i want to downgrade (separated by a newline). but somehow that list is not converted properly into a space separated list, which pacman can interpret. therefore, pacman gives me the following error (when i try to roll-back the update of "kwrite" and "ktexteditor"):

Oops! I forgot to edit that part. You need to edit all pacman commands (in func_b of course):

sudo pacman -Rdd "$pkgR" --color always --noconfirm

into :

sudo pacman -Rdd ${pkgR} --color always --noconfirm

This should work :

~/RAT's workspaces >>> ./pacui.bak                                                                                                                                    
loading packages...
warning: downgrading package artools-base (0.5.4-1 => 0.5.3-1)
warning: downgrading package artools-pkg (0.5.4-1 => 0.5.3-1)
resolving dependencies...
looking for conflicting packages...

Package (2)   Old Version  New Version  Net Change

artools-base  0.5.4-1      0.5.3-1        0.00 MiB
artools-pkg   0.5.4-1      0.5.3-1        0.00 MiB

Total Installed Size:  0.18 MiB
Net Upgrade Size:      0.00 MiB

:: Proceed with installation? [Y/n] 

Just opened a pull request for this #15


are you talking about this file: /tmp/pacui-packages-group

as far as i can see it, it is not security critical, because it is piped into fzf and then stored in a variable. only this variable is fed into pacman with sudo privileges.

am i wrong?

But don't they get removed afterwards? What if they want to run func_r again? Does the file gets replaced or will they gets added from the output?

excalibur1234 commented 7 years ago

But don't they get removed afterwards? What if they want to run func_r again? Does the file gets replaced or will they gets added from the output?

func_r is special, but take a look at the func_t, func_l, or func_d. there the file /tmp/pacui-packages-local is created and not removed afterwards. this file contains a list of all packages from the system repository and all locally installed packages and is only created when it does not exist yet. when it already exists, it is simply reused as is and not changed. it never gets recreated, when it already exists, just overwritten as seen by the > symbol in this code line: pacman -Slq | tr -d " " > /tmp/pacui-packages-local this is done because of performance reasons and should be noticeable by the user in func_i (where a list of all AUR pacakges gets downloaded).

now, that i think about it, /tmp/pacui-packages-local should be overwritten with the latest list of packages, because the user could have installed packages from the AUR since first using func_t, func_l, or func_d.

do you have other suggestions about the usage of a temporary file here? maybe, we should discuss this in a new issue?

excalibur1234 commented 7 years ago

one more comment: people using arch linux only have the AUR package to install pacui. therefore, i think we should stop doing development work (with breakages) in the master branch. i have created a devel branch for this kind of stuff. when we want people to test our stuff, we can move it over to master branch.

you can clone the development branch with (the single-branch argument is optional): git clone -b devel --single-branch https://github.com/......../pacui.git

thefallenrat commented 7 years ago

Thanks for replies and for merging this!

func_r is special, but take a look at the func_t, func_l, or func_d. there the file /tmp/pacui-packages-local is created and not removed afterwards. this file contains a list of all packages from the system repository and all locally installed packages and is only created when it does not exist yet. when it already exists, it is simply reused as is and not changed. it never gets recreated, when it already exists, just overwritten as seen by the > symbol in this code line:

pacman -Slq | tr -d " " > /tmp/pacui-packages-local

this is done because of performance reasons and should be noticeable by the user in func_i (where a list of all AUR pacakges gets downloaded).

now, that i think about it, /tmp/pacui-packages-local should be overwritten with the latest list of packages, because the user could have installed packages from the AUR since first using func_t, func_l, or func_d.

do you have other suggestions about the usage of a temporary file here? maybe, we should discuss this in a new issue?

Maybe we can do function-wide (?) $pacui_packages_local that set up said file with mktemp command, so that if it gets called again from the same or different func it can reuse it maintaining performance, and of course, removed it upon exit. But doing this requires user have to set up pacui-packages-local everytime they launch pacui....

And yeah I guess we should discuss this in a new issue

people using arch linux only have the AUR package to install pacui. therefore, i think we should stop doing development work (with breakages) in the master branch. i have created a devel branch for this kind of stuff. when we want people to test our stuff, we can move it over to master branch.

Oh cool! Should I have a future pull request, I will do this on devel branch!