desired features / to do list

[jrmarino]

1. Sanitize synthexec (issue #52)
2. re-enable watchdog (use new reapers in DragonFly and FreeBSD
3. Support clustering over multiple servers *
4. (long term) - implement 2 passes (including status check) option where the first pass rebuilds only ports that have no dependencies and the second pass does the recursive deleting. This will help when something like python is bumped (only python would be rebuilt and pkg wouldn't bring in rebuilt packages with the current scheme anyway). Requires lots of testing and needs to be the non-default option.
5. use popen with "re", not "r", but requires makefile changes for DF 4.4 and FreeBSD 9
6. web interface similar to poudriere
7. See if unnecessarily build depends can be skipped in "fetch prebuilt binaries" mode (https://forums.freebsd.org/threads/58080/#post-332342)
   • due to NFS sync races, the slave server must SCP package back to master server as part of the build cycle. The master server is the one that moves it to the package directory.

[jrmarino]

also remove mount of procfs for rust, it's going to be fixed for DF 46 and later.

[roker]

It would be really nice if port could be added to the building/updating but wouldnt be installed on a system. This comes quite handy for updating jails, it can be done by scripts but i would rather have an integrated solution instead.

[jrmarino]

I think you must not fully understand how Synth works. It only builds ports; none are installed on the system. There is a command or two that will call the pkg(8) program to install programs after they are built but nothing is installed as a result of building. Please realize that synth is a repository builder and you should update jails with a repository.

[Datapanic]

Please make it possible for synth to do syslog logging. In other words, if I want synth to log on local3.info, then it will do that. Various log levels would be great as well. See OpenLDAP's implementation of log levels for an idea. syslog logging would also allow synth's logs to go to a remote log server.

[debdrup]

Is there a specific reason why synth can't/doesn't offer to specify options for ports the first time they're built? That's one of the main features of portmaster that I like (ie. not having to run make config-recursive multiple times on first build), but since the status of portmaster is uncertain, it'd be nice if synth got it. Is it complexity of implementation, out-of-scope or something else that I'm missing? Assuming there are no reasons for not doing it, can I request it to be added at some point?

EDIT: That's a pretty sensible reason. Thanks for explaining it!

[jrmarino]

There's no need to run make config-recursive. You've implied that it's a need but it's not. The fact is that out of 26,000 ports, only a handful need to be customized. Most ports are DOC/NLS/EXAMPLES stuff that nobody changes from the default.

specific reason: it's a pain-in-the-butt extra step that most people agree is not needed. Back in the portupgrade days, forgetting to set an option actually broke the build (usually 10 minutes into what should have been an overnight run). Any changed option needs real understanding of the consequences and lacking that, the defaults should be kept. Someone should be able to look at the options directory and know that any directory listed there is a non-default option port and generally there are not many of those.

config-recursive is evil. It forces you to save options that are the same as the default. the Tools/scripts/redundant-opt-files.sh should be used to identify these and actually remove them. It's common that people have well over a 100 redundant saved options that can be deleted.

[sporkman]

I noticed that while setting things up, I quite often found myself sitting somewhere in the ports tree, usually because I've just run "make config" somewhere. If I then try to run synth, I get this error message:

[root@nac1 /usr/ports/www/apache24]# synth build www/apache24 Please change the current directory; Synth is unable to launch from here.

I kind of get what's going on here, but it sure would be nice if this restriction weren't necessary.

[debdrup]

If you just need to use make config on a port, you can do make -C /path/to/port config

[daemonblade]

Would it be possible to either

1. configure the watchdog timeouts or
2. enable/disable the hangmonitor

With crossbuilding, the host execution is dramatically slowed, and I'd like to be able to either bump up the "build" timeout or disable the hangmonitor completely when building some selected ports, eg java/openjdk8. The problematic ports tend to run for extended periods of time without generating output.

[jrmarino]

daemonblade@, Your best bet is to add a local patch that changes this line: https://github.com/jrmarino/synth/blob/master/src/portscan-buildcycle.adb#L573 specifically, set hangmonitor to "False".

[daemonblade]

Yup, I've got a local version of synth that already does that! I looked up some posts that you made previously with regard to watchdog timeouts. I was just wondering whether it'd be a good idea to expose the knobs in a config file or not.

[jrmarino]

I'd rather not expose it. There's only been one issue recently involving a worst-case scenario of ancient hardware coupled with a mechanical dist during the "depends" phase so we lengthened the timeout to address it. Crossbuilding on qemu is a whole different ball of wax. On one hand, there's probably a stronger need for a watchdog due to emulator use, but on the other hand, the timeout periods are inappropriate. I'm not sure yet what the best approach is, so for the (very few) people that crossbuild, I'd stick with a local patch for now.

At some point maybe an option can be added to the port itself to "sed" in the change or use an "extra" patch so it's easy to build that way. Or maybe we have the watchdog off by default and the option would turn it on. It seems that on a stock ports tree it probably doesn't come into play and only ports developers would really benefit from the watchdog.

[monwarez]

Adding the possibility to just fetch all distfiles required for upgrade

[jrmarino]

@daemonblade , If you want to propose a change to the port to give it an option to build with the watchdog on (off by default), you have my blessing. I am not the port's maintainer any more, but such a proposal would not irritate me. You would implement it with a post-patch-WATCHDOG-off: target I think.

[roker]

Automatic removal of NO_PACKAGE by some synth configuration option or something. The packages I am building are not meant for distribution and as such are not breaking the license but this is still stopping the packages to be built. It is forcing me each time to change Makefile for specific port and it is annoying.

[jrmarino]

roker@, you add "FORCE_PACKAGE" definition to your profile make.conf. It's not a synth issue, it's a ports issue.

edit:

FORCE_PACKAGE=    yes

[roker]

I knew this is a ports issue but I didnt found a workaround. Thank you very much...

[qcybb]

It is possible to add a new "synth force-upgrade" command?

Have Synth always record the last sessions build in to a "build.list" file. This new command would then read this file and perform a "pkg upgrade -f" on all packages contained in the build.list file.

This would allow us to reinstall all packages that were compiled/recompiled during the last session with ease - regardless whether PKG feels it's warranted or not.

[jrmarino]

There's no point. "synth force " does exactly this. the pkg(8) program is not involved in any way, so there seems to be some confusion on your part here. Synth => builds a repository pkg => installs packages from that repository. You can call "pkg install --force -r Synth" etc Truthfully, I'm starting to regret the convenience commands because users don't seem to recognize which tool is which doing which job due to the obscurity.

to really clarify, there's not point in "synth force" at all. If you want to force pkg to reinstall the packages, send the appropriate command directly to pkg to do so. Having synth rebuild packages pointlessly isn't going to make a difference. It already rebuilds aggressively if it needs to do it.

[qcybb]

I believe you misunderstood what I was asking. I'm not looking for you to change anything to do with how synth builds/rebuilds the repository.

I'm looking synth to create a "build.list" file of all packages that were added/upgraded in the repository during last session. Then I was hoping a new synth command that could read this "build.list" file and have synth issue a "pkg upgrade -f" on all items contained in that list (since PKG does not have file read support for this purpose).

[jrmarino]

were added/upgraded in the repository

What does that even mean? A repository is generated based on what packages are present in the /All directory. Every time you tell pkg(8) to generate a repository, it destroys all existing metadata, catalogs, and digests, then recreates them from scratch.

So you don't "add" packages to a repository. There's no way of tracking what is "upgraded" from a previous version of the repository.

So, yes, I guess I'm completely misunderstanding what you're suggesting. Nor do I understand what the point would be. A "partial force install"?
Don't forget that a lot of rebuilt packages are build dependencies, which are probably not installed.

From what i can tell, this is a request to be able to "force install" a portion of the entire installed packages rather than "force install" every single package. And that would be different from regular upgrades for ports where the maintainer failed to bump (maybe, I think pkg might be able to catch most of those too)

I guess you could do this with hooks. At the start of the build, create a blank file When each port completes, add the origin or package name to the list. At the end of the run, your list contains every port build during that run. See the hooks section of the man page.

[jrmarino]

don't forget that you can have many build runs between each repository generation. I don't see how you can easily track differences between those generations.

[qcybb]

Sorry for the confusion, all I really wanted was synth to create a file just like "/var/log/synth/01_success_list.log" - however with the timestamp removed and make the list space-delimited instead (print/freetype2 graphics/php71-gd).

That way I could use PKG to easily force upgrade the packages that synth last successfully built. If the list could exclude packages that are build dependencies (which are not installed) - even better.

[jrmarino]

well, first, an awk script could easily process 01_success_list.log (if you don't want to use the hooks) but problems with your idea: 1) the built list does not equal install list. pkg(8) will be confused if you try to force upgrade packages that aren't installed (think build dependencies) 2) it assumes a single run. Yes, typically there is only one run, but there could be multiple runs (say a server was done and fetching failed, causing many skips). Then the success list is not representative of what was built since it's redone for each run.

[qcybb]

The hooks worked out great.

I only have one build dependency so far, which I have on my "exclude list" - and can add additional entries to it later if needed. I would of course never do a "force upgrade" if everything did not build in a single run. The only thing I have fully automated is creating a new, clean build.list file on each run.

Thank you very much for your assistance on this.

[qcybb]

When using "synth status category/port", is it possible to have it display whether the port is just a build dependency or not?

For example :

$ synth status mail/spamassassin
These are the ports that would be built ([N]ew, [R]ebuild, [U]pgrade):
  N => net/p5-IO-Socket-IP
  N => security/p5-Digest-HMAC
  N => dns/p5-Net-DNS (Build Only)
  N => devel/p5-TimeDate

[jrmarino]

It's not possible. A port could be on the list twice, once as a build dependency and the next time as a run dependency. It's impossible to determine.

In other words, the list is like "sort unique". a build list could be 100,000 or longer, but most are duplicates and they are compressed together.

Then there's the philosophical aspect of "why is that information even needed?". The port needs building not matter if it gets installed or not.

[qcybb]

It's just for informational purposes only. Anything that needs to be installed uses more disk space.

[ghost]

Hi.

I am wondering if it would be possible to redistribute the total number of jobs of idle/shutdown slots.

I mean, often while building, at some point we get some idle or shutdown slots. I currently have 4 builds with max 8 jobs per builder, 32 jobs in total.

If (i.e.) I have just two slots building I got a maximum 16 jobs out of 32 possible ones. If those 16 possible (but not used) jobs could be redistributed to the builders what are actually building would be great, faster builds.

Thanks!

[jrmarino]

@lebarondemerde, it's not possible. For starters, there's a maximum jobs per port setting that would be violated. Secondly, you can't dynamically change the number of jobs during a build. You'd have to set it before the port build starts. Very easily all the slots could be used before a long build, but then they open up. You can't "add" jobs in the middle of a long build.

The other thing to point out is diminishing returns. Often adding another job to a build doesn't improve build times linearly. (in other words, moving from -j4 to -j5 probably won't reduce build time by 20%)

[ghost]

@jrmarino

I got it. Thanks!

[ghost]

https://github.com/jrmarino/synth/issues/53#issuecomment-293621998 @marino,

I see that synth 1.70 in FreeBSD default to watchdog disabled, while may be good to have the option exposed, disabling the watchdog by default is definitely a bad idea that lead to builders hangs forever when some port fail to build..

Considering your previous answers on the subject, on FreeBSD forum, I'm not sure you really want a watchdog disabled by default.

Please, leave the option but restore a "sane" default (watchdog enabled). Thanks.

[jrmarino]

well, I would have had it enabled by default, but it's not my call anymore. I'm not the port's maintainer.

[ghost]

Thanks for confirming, I got in touch with @ericbsd (current synth maintainer) and he will change the watchdog option to "enabled" in freebsd ports.

[qcybb]

Is it possible to extend the "status" option so it only outputs text if there are ports that need to be built?

Perhaps something like "synth status cron". That way we can add it to cron, and will only get emailed if something needs to be done.

I know we could use "pkg version", but that doesn't show if any "build only" ports require updating.

[driesmp]

Is it possible to make the command "synth just-build" default to LiveSystem-ports.conf (or something of your chosing like LiveSystem-(just)build.conf). At the moment, we need to provide an argument for just-build.

[jrmarino]

by the way, this ticket was always meant a person to-do list, not a wish list. If somebody wants to suggest a new feature, it should go in its own (well-titled) issue. Basically my to-do list got hijacked.

[ermo]

@jrmarino :

Re. 3. Support clustering over multiple servers *

What are your thoughts on "simply" adding support for mozilla's sccache?

To the best of my knowledge, it supports both C, C++ and Rust distributed compilation.