Plotman v0.5 on mac osx creates new plot every 20 secs

[S1m0nP1]

Upgraded to v0.5 on macOS Mojave in order to to add 'pool_contract_address:' to plots.

Previous version worked fine. Now starting the plot loop results in a new plot (and therefore new chia instances) being generated every 20 seconds until my machine runs out of resources and grinds to a halt!

# Default/example plotman.yaml configuration file

# https://github.com/ericaltendorf/plotman/wiki/Configuration#versions
version: [2]

logging:
        # One directory in which to store all plot job logs (the STDOUT/
        # STDERR of all plot jobs).  In order to monitor progress, plotman
        # reads these logs on a regular basis, so using a fast drive is
        # recommended.
        plots: /Users/simonperfitt/chialogs
        # transfers:
        # application:

# Options for display and rendering
user_interface:
        # Call out to the `stty` program to determine terminal size, instead of
        # relying on what is reported by the curses library.   In some cases,
        # the curses library fails to update on SIGWINCH signals.  If the
        # `plotman interactive` curses interface does not properly adjust when
        # you resize the terminal window, you can try setting this to True. 
        use_stty_size: True

# Optional custom settings for the subcommands (status, interactive etc)
commands:
        interactive:
                # Set it to False if you don't want to auto start plotting when 'interactive' is ran.
                # You can override this value from the command line, type "plotman interactive -h" for details
                autostart_plotting: True
                autostart_archiving: True

# Where to plot and log.
directories:
        # One or more directories to use as tmp dirs for plotting.  The
        # scheduler will use all of them and distribute jobs among them.
        # It assumes that IO is independent for each one (i.e., that each
        # one is on a different physical device).
        #
        # If multiple directories share a common prefix, reports will
        # abbreviate and show just the uniquely identifying suffix.
        tmp:
                - /Volumes/ssd/plots
                - /mnt/tmp/01
                - /mnt/tmp/02
                - /mnt/tmp/03

        # Optional: tmp2 directory.  If specified, will be passed to
        # chia plots create as -2.  Only one tmp2 directory is supported.
        # tmp2: /mnt/tmp/a

        # Optional: A list of one or more directories; the scheduler will
        # use all of them.  These again are presumed to be on independent
        # physical devices so writes (plot jobs) and reads (archivals) can
        # be scheduled to minimize IO contention.
        # 
        # If dst is commented out, the tmp directories will be used as the
        # buffer.
        dst:
                - /Volumes/new/plots

# Archival configuration.  Optional; if you do not wish to run the
# archiving operation, comment this section out.  Almost everyone
# should be using the archival feature.  It is meant to distribute
# plots among multiple disks filling them all.  This can be done both
# to local and to remote disks.
#
# As of v0.4, archiving commands are highly configurable.  The basic
# configuration consists of a script for checking available disk space
# and another for actually transferring plots.  Each can be specified
# as either a path to an existing script or inline script contents.
# It is expected that most people will use existing recipes and will
# adjust them by specifying environment variables that will set their
# system specific values.  These can be provided to the scripts via
# the `env` key.  plotman will additionally provide `source` and
# `destination` environment variables to the transfer script so it
# knows the specifically selected items to process.  plotman also needs
# to be able to generally detect if a transfer process is already
# running.  To be able to identify externally launched transfers, the
# process name and an argument prefix to match must be provided.  Note
# that variable substitution of environment variables including those
# specified in the env key can be used in both process name and process
# argument prefix elements but that they use the python substitution
# format.
#
# Complete example: https://github.com/ericaltendorf/plotman/wiki/Archiving
archiving:
  target: local_rsync
  env:
    command: rsync
    site_root: /farm/sites

# Plotting scheduling parameters
scheduling:
        # Run a job on a particular temp dir only if the number of existing jobs
        # before [tmpdir_stagger_phase_major : tmpdir_stagger_phase_minor]
        # is less than tmpdir_stagger_phase_limit.
        # Phase major corresponds to the plot phase, phase minor corresponds to
        # the table or table pair in sequence, phase limit corresponds to
        # the number of plots allowed before [phase major : phase minor].
        # e.g, with default settings, a new plot will start only when your plot
        # reaches phase [2 : 1] on your temp drive. This setting takes precidence
        # over global_stagger_m
        tmpdir_stagger_phase_major: 2
        tmpdir_stagger_phase_minor: 1
        # Optional: default is 1
        tmpdir_stagger_phase_limit: 1

        # Don't run more than this many jobs at a time on a single temp dir.
        tmpdir_max_jobs: 3

        # Don't run more than this many jobs at a time in total.
        global_max_jobs: 1

        # Don't run any jobs (across all temp dirs) more often than this, in minutes.
        global_stagger_m: 30

        # How often the daemon wakes to consider starting a new plot job, in seconds.
        polling_time_s: 20

        # Optional: Allows the overriding of some scheduling characteristics of the
        # tmp directories specified here.
        # This contains a map of tmp directory names to attributes. If a tmp directory 
        # and attribute is not listed here, the default attribute setting from the main 
        # configuration will be used
        #
        # Currently support override parameters:
        #     - tmpdir_stagger_phase_major (requires tmpdir_stagger_phase_minor)
        #     - tmpdir_stagger_phase_minor (requires tmpdir_stagger_phase_major)
        #     - tmpdir_stagger_phase_limit
        #     - tmpdir_max_jobs
        tmp_overrides:
                # In this example, /mnt/tmp/00 is larger and faster than the
                # other tmp dirs and it can hold more plots than the default,
                # allowing more simultaneous plots, so they are being started
                # earlier than the global setting above.
                "/mnt/tmp/00":
                        tmpdir_stagger_phase_major: 1
                        tmpdir_stagger_phase_minor: 5
                        tmpdir_max_jobs: 5
                # Here, /mnt/tmp/03 is smaller, so a different config might be
                # to space the phase stagger further apart and only allow 2 jobs
                # to run concurrently in it
                "/mnt/tmp/03":
                        tmpdir_stagger_phase_major: 3
                        tmpdir_stagger_phase_minor: 1
                        tmpdir_max_jobs: 2

# Plotting parameters.  These are pass-through parameters to chia plots create.
# See documentation at
# https://github.com/Chia-Network/chia-blockchain/wiki/CLI-Commands-Reference#create
plotting:
        # Your public keys.  Be sure to use the pool contract address for
        # portable pool plots.  The pool public key is only for original
        # non-portable plots that can not be used with the official pooling
        # protocol.
        # farmer_pk: ...
        # pool_pk: ...
        pool_contract_address: xch1pn3qnu8jw8ma99s734n3qxr4rqyfda7t80gvv0c78e27mef428sqqgg0xj

        # If you enable Chia, plot in *parallel* with higher tmpdir_max_jobs and global_max_jobs
        type: chia
        chia:
                # The stock plotter: https://github.com/Chia-Network/chia-blockchain
                k: 32                # k-size of plot, leave at 32 most of the time
                e: False             # Use -e plotting option
                n_threads: 2         # Threads per job
                n_buckets: 128       # Number of buckets to split data into
                job_buffer: 3389     # Per job memory

        # If you enable madMAx, plot in *sequence* with very low tmpdir_max_jobs and global_max_jobs
        madmax:
                # madMAx plotter: https://github.com/madMAx43v3r/chia-plotter
                n_threads: 4          # Default is 4, crank up if you have many cores
                n_buckets: 256        # Default is 256

[altendky]

Does plotman detect that the processes are running and report them? Such as via plotman interactive or plotman status. Also, what is the output of ps aux | grep 'plots create' when one or more of the plot processes have launched?

[S1m0nP1]

plotman interactive shows jobs as zero but reports and creates multiple log files (one for each job)

[S1m0nP1]

output of ps aux | grep 'plots create' is:

Last login: Tue Jul 13 14:09:23 on ttys000
simons-MBP:~ simonperfitt$ ps aux | grep 'plots create'
simonperfitt       530 187.2 30.1  8025816 2528588   ??  RNs   2:02pm  85:26.12 chia plots create -k 32 -r 2 -u 128 -b 3389 -t /Volumes/ssd/plots -d /Volumes/new/plots -c xch1pn3qnu8jw8ma99s734n3qxr4rqyfda7t80gvv0c78e27mef428sqqgg0xj
simonperfitt       951   0.0  0.0  4268040    792 s000  S+    3:04pm   0:00.00 grep plots create
simons-MBP:~ simonperfitt$ 

[S1m0nP1]

In the above example, I killed all running chia instances except one (otherwise my machine would crash before I could get to the terminal)

[altendky]

It looks like maybe there's a regression on handling chia installations done with the binary installer. I'll dig into it later.

[S1m0nP1]

Ok, thanks!

[altendky]

I think I've got a fix over in https://github.com/ericaltendorf/plotman/pull/865. This was also reported over in https://github.com/ericaltendorf/plotman/discussions/849. If you want to try it out you can use the regular installation instructions but with altendky/plotman@fix_chia_binary_process_detection instead of ericaltendorf/plotman@main. (in case someone passes by later, this branch will be deleted once merged)

[S1m0nP1]

reinstalled using: altendky/plotman@fix_chia_binary_process_detection. plotman status returns this error:

simons-MBP:~ simonperfitt$ plotman status
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.9/bin/plotman", line 8, in <module>
    sys.exit(main())
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/plotman/plotman.py", line 219, in main
    jobs = Job.get_running_jobs(cfg.logging.plots)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/plotman/job.py", line 227, in get_running_jobs
    parsed_command = parse_chia_plots_create_command_line(
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/plotman/job.py", line 86, in parse_chia_plots_create_command_line
    for argument in all_command_arguments
UnboundLocalError: local variable 'all_command_arguments' referenced before assignment

[altendky]

For what it's worth, in another PR I'm rewriting all this and adding decent test coverage so hopefully in the future we can avoid these regressions...

Go ahead and try installing again. I have added some more changes to that branch. Thanks for your time working through this.

[S1m0nP1]

yes!, that's fixed it. thanks for dealing with it so promptly