search

DragonFlyBSD / DPorts

The dedicated application build system for DragonFly BSD
Other
91 stars 43 forks source link

Default pkg.conf file has not been customized for DragonFly #39

Closed ftigeot closed 11 years ago

ftigeot commented 11 years ago

The default pkg.conf.sample file installed by ports-mgmt/pkg still references FreeBSD server information.

It would be nice if it could be modified to include a configuration suitable with DragonFly binary package mirrors instead.

Minimum information I have in mind:

# DragonFly (odd) development versions have a default ABI value set to the next future
# stable (even) version.
# The ABI directive allows to force the use of packages built for a different DragonFly
# version such as DragonFly 3.4 packages on DragonFly 3.5.
#ABI: dragonfly:3.4:x86:64

# Default Dports packaging mirror
#PACKAGESITE: http://mirror-master.dragonflybsd.org/dports/${ABI}/RELEASE
PACKAGESITE: http://mirror-master.dragonflybsd.org/dports/${ABI}/LATEST

It would also be useful if ports-mgmt/pkg installed the pkg.conf file if no previous version was present, so as to be able to use pkg(8) out of the box after installation

jrmarino commented 11 years ago

I agree the sample pkg.conf needs to be customized for DragonFly I think ABI suggestion is okay, although I would add the :32 version as well on a second commented line, so the user only has to uncomment the line for D32 rather than edit it (and uncomment it)

fupjack commented 11 years ago

Will this be doable through the ports-mgmt/pkg port or do we need to commit something to the 3.4 branch?

(asking because I'm hoping to build the release as soon as tonight, if the packages have completed upload by then.)

jrmarino commented 11 years ago

I do not see this as a ports issue. It's part of nrelease, right? The same thing that must ensure "pkg" is part of the release. It is critical that pkg and pkg.conf be included but this isn't a dports thing. Talk to swildner if in doubt.

fupjack commented 11 years ago

pkg gets built and installed since it's a dependency of all items in dports

On Thu, Apr 18, 2013 at 11:45 AM, jrmarino notifications@github.com wrote:

I do not see this as a ports issue. It's part of nrelease, right? The same thing that must ensure "pkg" is part of the release. It is critical that pkg and pkg.conf be included but this isn't a dports thing. Talk to swildner if in doubt.

— Reply to this email directly or view it on GitHubhttps://github.com/jrmarino/DPorts/issues/39#issuecomment-16584594 .

jrmarino commented 11 years ago

You're missing the big picture. You can't use any of the pre-built 19,500+ packages unless pkg is on the system, because that's what downloads the catalog. We are not requiring people to install a 25Mb+ set of dports in order to build a single "pkg" package so they can then use the prebuilt binaries (that's inane). We decided that all images must come with pkg preinstalled, just like they come with pkgsrc bootstrapped. swildner is very aware of this and agrees with it.

Not having pkg come with the system defeats the purpose of prebuilt repo. FreeBSD has a way of bootstrapping pkg according to their handbook but we don't. If we don't provide this with the release, it will be very conspicuous omission.

ftigeot commented 11 years ago

fupjack wrote:

pkg gets built and installed since it's a dependency of all items in dports

  • so does it have to be included? If both pkg and pkg.conf can be kept within the port, it makes life easier - nobody gets dports material until they actually take positive action to do so.

pkg.conf.sample is installed by the pkg port; having a DragonFly-customized pkg.conf file is thus a dport issue.

The only thing the release images would need to include to get working binary packages support out of the box is pre-installed pkg packages.

Francois Tigeot

jrmarino commented 11 years ago

hmmm? 1) it installs /usr/local/etc/pkg.conf.sample, not /usr/local/etc/pkg.conf. No port installs .conf files because it could overwrite an existing, manually created file. 2) You're the one that pointed out (correctly) that we needed a /usr/local/etc/pkg.conf for the release 3) nrelease needs exactly 2 things: the pkg.conf and the entire plist worth of "pkg" installed at prefix /usr/local.

maybe it was a typo, but it is not the job of ports to preinstall "customized" .conf files. Not a dports issue.

ftigeot commented 11 years ago

On Thu, Apr 18, 2013 at 11:16:13AM -0700, jrmarino wrote:

hmmm? 1) it installs /usr/local/etc/pkg.conf.sample, not /usr/local/etc/pkg.conf. No port installs .conf files because it could overwrite an existing, manually created file. 2) You're the one that pointed out (correctly) that we needed a /usr/local/etc/pkg.conf for the release 3) nrelease needs exactly 2 things: the pkg.conf and the entire plist worth of "pkg" installed at prefix /usr/local.

maybe it was a typo, but it is not the job of ports to preinstall "customized" .conf files. Not a dports issue.

The port installs a .conf.sample file; what's wrong with installing the same file with its regular .conf name if no previous copy is present ? One of pkgsrc or FreeBSD ports has a mechanism to handle this automatically if I'm not mistaken.

There's also another issue lurking here: the pkg.conf.sample file is taken verbatim from FreeBSD and includes a FreeBSD-specific repository URL.

I'm talking about this line: PACKAGESITE : http://pkg.freebsd.org/${ABI}/latest

Francois Tigeot

fupjack commented 11 years ago

On Thu, Apr 18, 2013 at 2:07 PM, jrmarino notifications@github.com wrote:

You're missing the big picture.

OK - I get it now. I had built from source before without having to think of where to get pkg, so it seemed perfectly easy to me. It would at least work, but I agree that it would be nicer to have all-binary really mean all-binary.

I assume we'd need to add pkg to the nrelease process; I don't know if swildner had a chance to try that yet. Going from the other emails on this, we'd want the pkg.conf.example file to have the DragonFly-specific path in there, I assume? We'd need to have $ABI set correctly for 32 bit or 64 bit, too.

ftigeot commented 11 years ago

jrmarino wrote

I agree the sample pkg.conf needs to be customized for DragonFly I think ABI suggestion is okay, although I would add the :32 version as well on a second commented line, so the user only has to uncomment the line for D32 rather than edit it (and uncomment it)

Proposed modification

--- pkg.conf.sample.orig    2013-03-31 01:52:51.000000000 +0100
+++ pkg.conf.sample 2013-04-18 21:01:37.249650000 +0200
@@ -2,11 +2,18 @@
  # For more information on the file format and
  # options please refer to the pkg.conf(5) man page

+# DragonFly (odd) development versions have a default ABI value set to the
+# next future stable (even) version.
+# The ABI directive allows to force the use of packages built for a 
different
+# DragonFly version such as DragonFly 3.4 packages on DragonFly 3.5.
+#ABI: dragonfly:3.4:x86:64
+#ABI: dragonfly:3.4:x86:32
+
  # Configuration options
-PACKAGESITE        : http://pkg.freebsd.org/${ABI}/latest
+PACKAGESITE: http://mirror-master.dragonflybsd.org/dports/${ABI}/LATEST
  #PKG_DBDIR        : /var/db/pkg
  #PKG_CACHEDIR     : /var/cache/pkg
-#PORTSDIR      : /usr/ports
+#PORTSDIR      : /usr/dports
  #PUBKEY           : /etc/ssl/pkg.conf
  #HANDLE_RC_SCRIPTS   : NO
  #PKG_MULTIREPOS       : NO
jrmarino commented 11 years ago

I think swildner may have already done it privately. I'm trying to find out on IRC right now.

fupjack commented 11 years ago

If swilder didn't get a chance to do this yet, how would we add it to the nrelease build process? I'm unsure how to bootstrap it.

ftigeot commented 11 years ago

fupjack wrote:

If swilder didn't get a chance to do this yet, how would we add it to the nrelease build process? I'm unsure how to bootstrap it.

The nrelease process itself is heavily tied with pkgsrc but it should be possible to add the pkg package in the final directory hierarchy.

pkg has a -c option to chroot in a specific directory

Something like pkg -c pkg-1.0.11.txz /path/to/release/dir before creating the images should do the trick.

Francois Tigeot

jrmarino commented 11 years ago

Hmm, we need to take care of this mirror issue in the next 24 hours. I guess the easiest, quickest way is to start with these dns entries bapt talked about?

ftigeot commented 11 years ago

jrmarino wrote:

Hmm, we need to take care of this mirror issue in the next 24 hours. I guess the easiest, quickest way is to start with these dns entries bapt talked about?

Is this discussion archived somewhere ? I fear I missed the most important bits...

Francois Tigeot

jrmarino commented 11 years ago

The main discussion was by bapt over IRC and it's not recorded (unfortunately). I thought http://mirror-master.dragonflybsd.org/dports/ wasn't active yet, but it is.

As it stands now, people only will see avalon and no other mirror, even if avalon fails. With the dns entries, it will try avalon first and then each listed mirror in the case of a failure. an http page could also list the mirrors. Or like pcbsd does, a dynamic http mirror page returns one 1 server, but it's been pre-selected by geolocation. This is probably the best although I don't know how often pkg checks that (once? once per day? per command line request?)

jrmarino commented 11 years ago

maybe tuxillo or luxh have a transcript -- the discussion is probably a week old

ftigeot commented 11 years ago

jrmarino wrote:

Or like pcbsd does, a dynamic http mirror page returns one 1 server, but it's been pre-selected by geolocation. This is probably the best although I don't know how often pkg checks that (once? once per day? per command line request?)

If it's really doing geo-location, it should redirect all http requests.

I don't know what PC-BSD is using but this piece of software seems to be doing exactly that: http://www.mirrorbrain.org/

Francois Tigeot

ftigeot commented 11 years ago

jrmarino wrote:

The main discussion was by bapt over IRC and it's not recorded (unfortunately). I thought http://mirror-master.dragonflybsd.org/dports/ wasn't active yet, but it is.

As it stands now, people only will see avalon and no other mirror, even if avalon fails. With the dns entries, it will try avalon first and then each listed mirror in the case of a failure. an http page could also list the mirrors. Or like pcbsd does, a dynamic http mirror page returns one 1 server, but it's been pre-selected by geolocation. This is probably the best although I don't know how often pkg checks that (once? once per day? per command line request?)

Having just discussed mirroring with bapt@ on #pkgng, I'm afraid we won't be able to use http mirroring as-is.

This feature currently works by:

Since all mirrors MUST use the same directory paths and the redirector host MUST return the mirror list from the / location, we can't use mirror-master.dragonflybsd.org and its mirrors in their existing configuration.

Francois Tigeot

jrmarino commented 11 years ago

It sound like you are the best position to recommend how we move forward. What needs to happen / what should the configuration be?

ftigeot commented 11 years ago

jrmarino wrote:

It sound like you are the best position to recommend how we move forward. What needs to happen / what should the configuration be?

  • Using DNS mirroring would mean changing dragonflybsd.org configuration, which introduces a whole new category of potential issues. I'll avoid it if possible
  • Pkgng's HTTP mirroring feature has been created too fulfill PC-BSD specific requirements and doesn't cut it in our case
  • The best way to implement automatic mirror redirection would be to modify pkgng and add a new mirroring feature. This is what bapt recommends.

We obviously won't be able to implement this and debug it properly for the DragonFly 3.4 release.

I suggest we choose a primary mirror reasonably suitable for the whole world by default, with continent-specific alternatives commented out in pkg.conf .

Francois Tigeot

jrmarino commented 11 years ago

Okay, given that I suggest that the pkg.conf that we provide with the release already be prefilled out with a commented list of mirrors, along with comments that indicate their physical location. The list doesn't have to be comprehensive.

That way users can easily switch from avalon to one closer to them by adding and removing hash characters. Let's make sure swildner sees this because he's planning to add pkg.conf to nrelease.

ftigeot commented 11 years ago

jrmarino wrote:

Okay, given that I suggest that the pkg.conf that we provide with the release already be prefilled out with a commented list of mirrors, along with comments that indicate their physical location. The list doesn't have to be comprehensive.

That way users can easily switch from avalon to one closer to them by adding and removing hash characters. Let's make sure swildner sees this because he's planning to add pkg.conf to nrelease.

One of the issues is we don't currently have dports mirrors.

I'm currently copying your x32 set from avalon to pkg.wolfpond.org; that would at least give us one server in Europe.

Francois Tigeot

jrmarino commented 11 years ago

what I meant was that we know the mirrors for pkgsrc, images, etc, and we can assume that once announced these mirrors will pick up dports too.

So I mean to anticipate as I am aware there are no mirrors today.

fupjack commented 11 years ago

There's a mirrors@dragonflybsd.org mailing list; I'm not sure how many of the mirrors have people on it, but there's 36 names on it, so it must be some. Is there any reason not to post about /dports there, to get the mirrors started?

On Fri, Apr 19, 2013 at 7:25 AM, jrmarino notifications@github.com wrote:

what I meant was that we know the mirrors for pkgsrc, images, etc, and we can assume that once announced these mirrors will pick up dports too.

So I mean to anticipate as I am aware there are no mirrors today.

— Reply to this email directly or view it on GitHubhttps://github.com/jrmarino/DPorts/issues/39#issuecomment-16648086 .

jrmarino commented 11 years ago

I can't think of a reason to delay mirroring at this point.

ftigeot commented 11 years ago

fupjack wrote:

There's a mirrors@dragonflybsd.org mailing list; I'm not sure how many of the mirrors have people on it, but there's 36 names on it, so it must be some. Is there any reason not to post about /dports there, to get the mirrors started?

I've sent a mail advertising the new http url on mirror-master.dragonflybsd.org.

Francois Tigeot

jrmarino commented 11 years ago

it's run it's course.