eix-sync fails to do its job

[pavlix]

I'm sorry if I don't describe the issue properly at first.

I am using this repository for portage git configuration and at the same time I'm using eix. It sort of worked for me until recently. Now I don't see main gentoo ebuilds in eix at all, only ebuilds from overlays. Also I'm seeing many modified files in git status and thus I assume rsync is probably used somewhere despite the repository is git. Also the metadata generation time (required I guess) is not anywhere near the couple of minutes suggested by online resources.

Any suggestions?

[pavlix]

Fork by @srcshelton seems to work correctly for me.

[hasufell]

I added @zx2c4 and @junghans as collaborators

[mgorny]

What version of Portage are you using? There were some changes wrt shallow fetching recently. Long story short, someone tried to force git fetch --depth 1 by default, then we attempted to fix new bugs resulting from it it several times, and finally reverted it all as impossible to support. I guess that could leave the repository in some kind of semi-damaged state.

Maybe try git reset --hard, and see what happens next. Some logs would be useful too.

[pavlix]

@mgorny I'm using a recently updated sys-apps/portage 2.3.0. But as I said with the fork it seems to work correctly. Should I retry with this repo's master?

[mgorny]

The only major difference between this repo and the fork I see is a changed location. Maybe the fork just caused you to re-clone in a new location?

[pavlix]

@mgorny I also spotted the location change. What is the purpose of the location change? Should I try to actually rm -rf /usr/portage and switch back? What should I keep in that case, just /usr/portage/distfiles so that I don't download things again?

[mgorny]

@pavlix, shouldn't you ask the fork owner? ;-) But /usr/portage is long considered a bad idea, /usr is no place to add custom runtime-writeable data to. You can use whichever paths are convenient for you, but it's a good idea to have distfiles split from repo.

And yes, you need to keep/move that to keep the downloaded files (it's DISTDIR in make.conf). If you built/fetched binary packages, there's also packages subdir (PKGDIR).

[srcshelton]

In all honesty, I'm pretty sure that @mgorny is spot-on: it's not anything I've changed, but the fact that the repo was cloned to a different location.

I'm not sure if it's due to force-pushes or something else, but the Gentoo repo frequently re-writes its history in ways which means that a simple 'git pull' won't work. I use the following script (as fixme.sh in the root of the repo - /var/db/repo/gentoo in my case, see below):

#! /bin/bash

cd "$( dirname "$( readlink -e "${0}" )" )" || exit 1
[[ -e .git ]] || exit 1

git status 2>/dev/null | grep -q '^\s\+modified:\s\+' || exit 0

git reset --hard HEAD

git status 2>&1 |
        grep $'^\t' |
        grep -Ev "^\s(.gitattributes|$( basename "${0}" ))$" |
        while read -r F
do
        if [[ -d "${F}" ]]; then
                rm -vr "${F}"
        else
                rm -v "${F}"
        fi
        rmdir -p "$( dirname "${F}" )" 2>/dev/null
done

... to remove any forked changes which would prevent a pull from succeeding, and then call this from cron.d/portage-sync as follows:

# Global variables
SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root
HOME=/

# Local variables
repo="$( portageq get_repo_path / "$( grep -m 1 'main-repo\s*=\s*' /etc/portage/repos.conf/* | cut -d'=' -f 2- | head -n 1 | sed 's/^\s\+// ; s/\s\+$// ' )" 2>/dev/null )"

#  Fields:
#   minute              (0-59)
#   hour                (0-23)
#   day of month        (1-31)
#   month of year       (1-12)
#   day of week         (0-6, Sunday == 0 & 7)

# Keep master portage tree in sync
30   3  * * *   root    test -d "${repo}" && chown -R portage:wheel "${repo}" && chmod -R ug+rwX "${repo}"
# The path immediately below is correct - git stores all submodule information in the parent metadata directory...
40   3  * * *   portage test -d "${repo}" && test -e "${repo}"/metadata/news/.git && pushd "${repo}" >/dev/null 2>&1 && git submodule update --remote --merge
41   3  * * *   portage test -x "${repo}"/fixme.sh && "${repo}"/fixme.sh

# Note differing 'throw away all output' vs. 'throw away normal messages but keep errors' below...
45   3  * * 2-7 root    test -x /usr/sbin/emaint && { /usr/sbin/emaint sync --auto >/dev/null 2>&1 ; } && { test -x /usr/bin/eix-update && /usr/bin/eix-update -H >/dev/null 2>&1 || true ; } && { test -x /usr/bin/emerge && /usr/bin/emerge -Duv --changed-use --with-bdeps y world -f >/dev/null 2>&1 ; }
# This stage takes over 13(!) hours to complete... :(
45   3  * * 1   root    test -x /usr/sbin/emaint && { test -e '/var/db/repo/gentoo/metadata/timestamp.x' && rm '/var/db/repo/gentoo/metadata/timestamp.x' ; true ; } && { /usr/sbin/emaint sync --auto >/dev/null 2>&1 ; } && { test -x /usr/bin/egencache && /usr/bin/egencache --repo gentoo --write-timestamp --update 2>&1 >/dev/null && /usr/bin/egencache --repo gentoo --update-use-local-desc --preserve-comments 2>&1 >/dev/null || true ; } && { test -x /usr/bin/eix-update && /usr/bin/eix-update -H >/dev/null 2>&1 || true ; } && { test -x /usr/bin/emerge && /usr/bin/emerge -Duv --changed-use --with-bdeps y world -f >/dev/null 2>&1 ; }

# vi: set nowrap:

I quick note on locations: I have systems where I want /usr to be local- and specific- to the host in question, but read-only unless performing a package upgrade. Having the portage tree in /usr/portage has always felt a little off-kilter, so I use the following directories instead:

• var/db/repo/gentoo - Gentoo Portage repo
• var/db/repo/<other> - Other overlays
• var/cache/portage/build - Portage temporary/build root
• var/cache/portage/dist - Portage distfiles
• var/cache/portage/local - NFS-mounted shared replacement for /usr/portage/local
• var/cache/portage/pkg - Binary package repo
• var/cache/portage/rpm - For the sake of completeness ;)

... which results in nice semantics where the contents of the cache directory may be considered ephemeral without breaking Portage, and there is no overlaying of directory usage - compared to the default configuration where /usr/portage also contains distfiles, packages and local, but not overlays.

I should probably document this better somewhere more permanent ;)

[srcshelton]

@mgorny Just beat me to the post by a fraction of a second there ;)

[mgorny]

@srcshelton, force pushes are forbidden (and blocked on gitolite) on ::gentoo. You're suffering from the --depth 1 fetching problem.

[srcshelton]

@mgorny Is there any (better?) fix for that right now?

[hasufell]

This is a bug in portage. Report it on bugzilla, this one also seems related. Alternatively, switch to paludis. For paludis, I also happen to have a gentoo-git-config.

[pavlix]

In all honesty, I'm pretty sure that @mgorny is spot-on: it's not anything I've changed, but the fact that the repo was cloned to a different location.

@srcshelton You mean you did change the location but apart from making a new clone your changes don't include a fix, right?

shouldn't you ask the fork owner? ;-) But /usr/portage is long considered a bad idea, /usr is no place to add custom runtime-writeable data to. You can use whichever paths are convenient for you, but it's a good idea to have distfiles split from repo.

@mgorny Is there any written best practice or something like that?

@hasufell Are you going to accept changes made by @srcshelton regarding gentoo repo location?

@srcshelton I'm using git pull --rebase just with any git repo without any issues. It's just the eix-sync or emerge --sync that seem to perform the sync the wrong way. I'm curious whether configuring gentoo in repos.conf is enough to disable the default rsync method. Your path changes look reasonable. It would be glad to document it and share it with people who could consider bringing those changes to official gentoo.

Thanks everyone for your comments, I hope we can get the information somewhere permanent as @srcshelton suggested so that they don't get lost when this issue gets closed.

[mgorny]

@mgorny Is there any written best practice or something like that?

Nope, just the never-ending bikeshed problem.

[junghans]

I see, I never came across a permission issue as I am using the patch from #16.