Recursive Packaging

[Korusuke]

As already done in the current PR, now args.packages is a list which upt iterates over and generates all the ports.

So using this list we can simply add a recursive feature like once the frontend returns the list of required packages. Add them to the list if not already in the list. But in this, there is no check or control by backend which can be added if instead of upt adding to list, the backend does that.

Now the question is in the backend should we "regenerate" ports if already present in MacPorts tree and second, currently if we output to file the program stop on encountering an existing file which would be a problem if we try recursively packaging ports having the same dependency not committed to MacPorts tree. So should we change this to just logging the error and continue?

[Steap]

So using this list we can simply add a recursive feature like once the frontend returns the list of required packages. Add them to the list if not already in the list. But in this, there is no check or control by backend which can be added if instead of upt adding to list, the backend does that.

How do we handle loops in our dependency graph? Especially since we are building it as we go?

Should we do a depth-first exploration or a breadth-first exploration?

Now the question is in the backend should we "regenerate" ports if already present in MacPorts tree

Probably not.

currently if we output to file the program stop on encountering an existing file which would be a problem if we try recursively packaging ports having the same dependency not committed to MacPorts tree.

Well, we're supposed to remember what we've packaged so far, aren't we?

[Korusuke]

How do we handle loops in our dependency graph? Especially since we are building it as we go?

Since we are building as we go we can simply check while adding the dependency if its already in the list or not.

Should we do a depth-first exploration or a breadth-first exploration?

Currently what I tried is breadth-first. Here is the code that I am using currently (can be cleaned and optimized further):

for pkg in args.package:
            try:
                upt_pkg = None
                upt.log.logger_set_formatter(logger, 'Frontend')
                upt_pkg = frontend.parse(pkg)
                upt_pkg.frontend = args.frontend
                upt.log.logger_set_formatter(logger, 'Backend')
                backend.create_package(upt_pkg, output=args.output)
                deps = []
                for i in upt_pkg.requirements.get('run',[]):
                    deps.append(i.name)
                for i in upt_pkg.requirements.get('test', []):
                    deps.append(i.name)
                for i in upt_pkg.requirements.get('build', []):
                    deps.append(i.name)
                for i in upt_pkg.requirements.get('config', []):
                    deps.append(i.name)
                for dep in deps:
                    if dep not in args.package:
                        args.package.append(dep)

currently if we output to file the program stop on encountering an existing file which would be a problem if we try recursively packaging ports having the same dependency not committed to MacPorts tree.

Well, we're supposed to remember what we've packaged so far, aren't we?

We can remember the packages of one-run but I am talking about a condition where the maintainer, for example, generates packages for two different package like upt-pypi and upt-cpan. So they both will be executed one after another and if we consider that requests is not present in the online macports-tree then we have a conflict here.

[reneeotten]

Thinking about this a bit more I am not sure whether we should add the "input" function at all and not just stick to "recursive packaging" only.

My two cents:

• maintainer supplies one package he/she wants to (recursively) package
• we figure out what the required dependencies are
• we check whether they are already present in MacPorts and, if so, if that version can fulfill the requirement, then there are three options:
  1. if so, we move on (possibly checking whether a newer version is available and logging that)
  2. if not and there is a newer version available on PyPI that does fulfill the requirement we update the package (here the "update feature" needs to be implemented)
  3. if we cannot find the required package on PyPI, we log this as an error and stop

[Steap]

Thinking about this a bit more I am not sure whether we should add the "input" function at all and not just stick to "recursive packaging" only.

I'm not against it, but I think people will usually package one thing at a time.

My two cents:

• maintainer supplies one package he/she wants to (recursively) package
• we figure out what the required dependencies are

The issue is that we need to do this as we go.

• we check whether they are already present in MacPorts and, if so, if that version can fulfill the requirement, then there are three options:

How should one do that? This is MacPorts specific, so you probably have a better idea about this than I do :)

Checking whether the version is right is still kind of tricky right now, so we'll probably skip this. The issue is that do not really have a unified format for version requirement. Let's just skip this for now.

1. if so, we move on (possibly checking whether a newer version is available and logging that)

OK.

2. if not and there is a newer version available on PyPI that does fulfill the requirement we update the package (here the "update feature" needs to be implemented)

With both "--recursive" and "--update" being experimental and complex, that is quite optimistic.

3. if we cannot find the required package on PyPI, we log this as an error and stop

Do we really want to fail if one package out of 90 cannot be found?

[reneeotten]

I agree with @Steap that people normally package one thing-at-the-time, so we need recursive packaging but I don't think we need the ability to specify multiple packages as input - so let's focus on the first!

I see the point that we figure out the dependencies as we go, but starting with all requirements for the actual thing we want to package and then go down their dependency-lists as @Korusuke is suggesting seems fine... correct?

Checking whether a package is present in MacPorts could be done with port info <packagename>. Of note, this doesn't necessarily mean that the correct sub-port (e.g., py37) is present, so for example for the Python ports one could also loop over the python.versions and check each of them specifically using port info pyXY-<packagname>.

The versioning stuff is important I think but if that's not possible yet we can skip it for now...

Sure, we indeed don't need to fail if one package cannot be found, let's just log it clearly ;)

[reneeotten]

Closed with 5847f0f628e37720ba2eceda8974e6a63f37f1ae