recursive packaging ends mid way

[Korusuke]

If a specific package is not available/not found upstream then recursive packaging stops midway and there's no automagic way to package the remaining packages.

Example: ghci package for hackage, there are many missing packages in hackage.

[reneeotten]

Hhm... didn't we talk about this in Issue #32 and concluded that we shouldn't stop the recursive packaging if one thing cannot be found by a frontend? So yes, I would say that this is an issue that indeed would need fixing... we probably should just clearly log such occurrences and try to continue.

In you're example you get the The package <packagenmae> could not be found by backend hackage message, so upt raised an InvalidPackageNameError in the package function.

The way I see it, that is supposed to happen when this is the case for the package_name you're explicitly giving to upt package -f hackage -b macports <package_name>. However, if it happens within the if recursive: block it should just be logged probably as an error and upt should try to continue. So perhaps it's just a matter of moving the if`` .....except```` stuff around a bit? @Steap do you agree with this assessment?

[Korusuke]

Hhm... didn't we talk about this in Issue #32 and concluded that we shouldn't stop the recursive packaging if one thing cannot be found by a frontend? So yes, I would say that this is an issue that indeed would need fixing... we probably should just clearly log such occurrences and try to continue.

Yeah right we did discuss this, just wanted to bring it up again as I faced this issue a lot while trying to recursively package haskell packages and ended up just changing the raiseFrontendError to a log message.

Just wanted to put it here to get your and @Steap's view on how we should move forward with this.

[reneeotten]

Yeah right we did discuss this, just wanted to bring it up again as I faced this issue a lot while trying to recursively package haskell packages and ended up just changing the raiseFrontendError to a log message.

I am not sure I understand that... if a package cannot be found by the backend that should raise an InvalidPackageNameError, correct? So which FrontendError did you change to only log a message (somewhere in upt or in the backend)? With your change does it then also only show a log message when not using the recursive flag (and, if so, is that what we want)?

Does what I wrote in the first comment make sense to you and what do you think is the best way/place in the code (i.e., in upt or the backend) to resolve this?

[Korusuke]

I am not sure I understand that... if a package cannot be found by the backend that should raise an InvalidPackageNameError, correct? So which FrontendError did you change to only log a message (somewhere in upt or in the backend)? With your change does it then also only show a log message when not using the recursive flag (and, if so, is that what we want)?

So sorry, I meant InvalidPackageNameError. Just Ignore my previous comment.

Does what I wrote in the first comment make sense to you and what do you think is the best way/place in the code (i.e., in upt or the backend) to resolve this?

Yes it makes sense and works too. I would say upt should resolve this as this is an error for all backends. Below changes in upt solves this issue.

try:
    upt_pkg = frontend.parse(pkg_name)
except upt.exceptions.InvalidPackageNameError as e:
    if args.recursive:
        logger.error(f'{pkg_name} was not found upstream')
        return
    else:
        raise upt.exceptions.InvalidPackageNameError(args.backend, pkg_name)

While working on this, I also stumbled upon another issue in upt-cpan wherein it fails even if the pod is available upstream...I have reported this issue here....I have not checked if cpan2port has same issue or not.

[reneeotten]

(please mention @Steap in discussions like this so that he gets notified as it's directly related to upt).

From this small code-snippet it's not clear to me where this exactly would go and/or what the surrounding code looks like. I am, for example, not convinced about the return part, the point is that we don't want to stop (or return) anything while doing a recursive packaging and when a package cannot be found. It might work, but please supply the full code of the package function and/or submit a PR to upt so that this can be properly discussed.

[Steap]

Please do as @reneeotten suggested. Upt has a modular design, and bugs should be reported to the right module. When in doubt, report it to upt/upt itself.