Plotting goes into the 20sec loop and opens bunch of plot works

[NicodemPL]

Hi, I got huge problem with launching properly my plotman. Once I use plotman plot got this strange behaviour:

...starting plot loop
...sleeping 20 s: (True, 'Starting plot job: chia plots create -k 32 -r 4 -u 128 -b 3400 -t /home/nicodem/Desktop/chia -d /media/nicodem/data/final ; logging to /home/nicodem/chialogs/2021-05-10-18:10:18.log')
...sleeping 20 s: (True, 'Starting plot job: chia plots create -k 32 -r 4 -u 128 -b 3400 -t /home/nicodem/Desktop/chia -d /media/nicodem/data/final ; logging to /home/nicodem/chialogs/2021-05-10-18:10:38.log')

And it goes to loop - starts new process every 20 seconds. I have to Ctrl-C this and got such info:

File "/home/nicodem/.local/bin/plotman", line 8, in <module>
    sys.exit(main())
  File "/home/nicodem/.local/lib/python3.8/site-packages/plotman/plotman.py", line 149, in main
    time.sleep(cfg.scheduling.polling_time_s)
KeyboardInterrupt

While ps ax | grep chia shows bunch of jobs started:

  15146 ?        SNs    0:00 chia plots create -k 32 -r 4 -u 128 -b 3400 -t /home/nicodem/Desktop/chia -d /media/nicodem/data/final
  15147 ?        SNLl   2:53 chia plots create -k 32 -r 4 -u 128 -b 3400 -t /home/nicodem/Desktop/chia -d /media/nicodem/data/final
  15484 ?        SNs    0:00 chia plots create -k 32 -r 4 -u 128 -b 3400 -t /home/nicodem/Desktop/chia -d /media/nicodem/data/final
  15485 ?        SNLl   2:24 chia plots create -k 32 -r 4 -u 128 -b 3400 -t /home/nicodem/Desktop/chia -d /media/nicodem/data/final
  15821 ?        SNs    0:00 chia plots create -k 32 -r 4 -u 128 -b 3400 -t /home/nicodem/Desktop/chia -d /media/nicodem/data/final
  15822 ?        SNLl   1:46 chia plots create -k 32 -r 4 -u 128 -b 3400 -t /home/nicodem/Desktop/chia -d /media/nicodem/data/final
  16159 ?        SNs    0:00 chia plots create -k 32 -r 4 -u 128 -b 3400 -t /home/nicodem/Desktop/chia -d /media/nicodem/data/final
  16160 ?        SNLl   1:17 chia plots create -k 32 -r 4 -u 128 -b 3400 -t /home/nicodem/Desktop/chia -d /media/nicodem/data/final
  16496 ?        SNs    0:00 chia plots create -k 32 -r 4 -u 128 -b 3400 -t /home/nicodem/Desktop/chia -d /media/nicodem/data/final
  16497 ?        SNLl   0:58 chia plots create -k 32 -r 4 -u 128 -b 3400 -t /home/nicodem/Desktop/chia -d /media/nicodem/data/final
  16833 ?        SNs    0:00 chia plots create -k 32 -r 4 -u 128 -b 3400 -t /home/nicodem/Desktop/chia -d /media/nicodem/data/final
  16834 ?        SNLl   0:50 chia plots create -k 32 -r 4 -u 128 -b 3400 -t /home/nicodem/Desktop/chia -d /media/nicodem/data/final
  17170 ?        SNs    0:00 chia plots create -k 32 -r 4 -u 128 -b 3400 -t /home/nicodem/Desktop/chia -d /media/nicodem/data/final
  17171 ?        SNLl   0:38 chia plots create -k 32 -r 4 -u 128 -b 3400 -t /home/nicodem/Desktop/chia -d /media/nicodem/data/final
  17509 ?        SNs    0:00 chia plots create -k 32 -r 4 -u 128 -b 3400 -t /home/nicodem/Desktop/chia -d /media/nicodem/data/final
  17510 ?        SNLl   0:28 chia plots create -k 32 -r 4 -u 128 -b 3400 -t /home/nicodem/Desktop/chia -d /media/nicodem/data/final
  17846 ?        SNs    0:00 chia plots create -k 32 -r 4 -u 128 -b 3400 -t /home/nicodem/Desktop/chia -d /media/nicodem/data/final
  17847 ?        SNLl   0:20 chia plots create -k 32 -r 4 -u 128 -b 3400 -t /home/nicodem/Desktop/chia -d /media/nicodem/data/final
  18183 ?        SNs    0:00 chia plots create -k 32 -r 4 -u 128 -b 3400 -t /home/nicodem/Desktop/chia -d /media/nicodem/data/final
  18184 ?        SNLl   0:14 chia plots create -k 32 -r 4 -u 128 -b 3400 -t /home/nicodem/Desktop/chia -d /media/nicodem/data/final
  18523 ?        SNs    0:00 chia plots create -k 32 -r 4 -u 128 -b 3400 -t /home/nicodem/Desktop/chia -d /media/nicodem/data/final
  18524 ?        SNLl   0:08 chia plots create -k 32 -r 4 -u 128 -b 3400 -t /home/nicodem/Desktop/chia -d /media/nicodem/data/final

Any idea what is wrong?

[altendky]

Try the installation instructions again with @development instead of @main. There are fixes that are likely related to your issue. Let us know if that solves it for you.

[NicodemPL]

Done reinstall - goes into stagger so looks promising now, thanks!

[NicodemPL]

Nope, still same issue. I had to restart this PC and even when I try to reinstall now into development - got same 20s issue.

[naterongi]

having the same issue

[altendky]

What do you get for plotman status output? Please share that along with the output of whatever you use to plot, interactive or plot, as well as ps aux | grep 'plots create'. Also explain what version you are using by way of how you installed it.

Have either of you installed chia using the .deb experimental GUI installer? If so then https://github.com/ericaltendorf/plotman/pull/418 may be relevant. You could try it out by following the regular readme installation but use git+https://github.com/altendky/plotman@skip_matching_plot_process_parents as the url.

Sorry for the trouble.

[naterongi]

Plotman status has the words categories written, but no plot being recorder under. Same as in the interactive menu, It tries to plot, but then disappears. I am a newby at plotting so i followed the interactive guide https://www.youtube.com/watch?v=cahPECwVN78, I did everything this guy did in his video, just I wasn't sure if my keys where correct or if I need them at all. I can plot using the gui, but every single time I try to use plotman it doesnt work

[NicodemPL]

Mine installation was done via .deb GUI experimental installer. All upgrades goes via this since then. I do not want to mess with another installation. Will try to poke with yours suggestions during weekend, currently plottin fully manual.

On Fri, May 14, 2021 at 5:48 AM Kyle Altendorf @.***> wrote:

What do you get for plotman status output? Please share that along with the output of whatever you use to plot, interactive or plot, as well as ps aux | grep 'plots create'. Also explain what version you are using by way of how you installed it.

Have either of you installed chia using the .deb experimental GUI installer? If so then #418 https://github.com/ericaltendorf/plotman/pull/418 may be relevant. You could try it out by following the regular readme installation but use git+ @.***_matching_plot_process_parents as the url.

Sorry for the trouble.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ericaltendorf/plotman/issues/316#issuecomment-840985592, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKYUMGMV543GD3STPCE5PO3TNSMRNANCNFSM44RTA7NA .

[slowbro]

Hey there,

I'm also seeing this issue. I installed chia from the instructions on their github. I am running FreeBSD 12.2p4, running chia in a jail; I am also using the development build as was suggested.

I did a small amount of digging around, but I wasn't able to find much. Here is my config:

Config

``` # Options for display and rendering user_interface: # Call out to the `stty` program to determine terminal size, instead of # relying on what is reported by the curses library. In some cases, # the curses library fails to update on SIGWINCH signals. If the # `plotman interactive` curses interface does not properly adjust when # you resize the terminal window, you can try setting this to True. use_stty_size: True # Where to plot and log. directories: # One directory in which to store all plot job logs (the STDOUT/ # STDERR of all plot jobs). In order to monitor progress, plotman # reads these logs on a regular basis, so using a fast drive is # recommended. log: /chiatmp/logs # One or more directories to use as tmp dirs for plotting. The # scheduler will use all of them and distribute jobs among them. # It assumes that IO is independent for each one (i.e., that each # one is on a different physical device). # # If multiple directories share a common prefix, reports will # abbreviate and show just the uniquely identifying suffix. tmp: - /chiatmp # Optional: Allows overriding some characteristics of certain tmp # directories. This contains a map of tmp directory names to # attributes. If a tmp directory and attribute is not listed here, # it uses the default attribute setting from the main configuration. # # Currently support override parameters: # - tmpdir_max_jobs #tmp_overrides: # In this example, /mnt/tmp/00 is larger than the other tmp # dirs and it can hold more plots than the default. #"/mnt/tmp/00": # tmpdir_max_jobs: 5 # Optional: tmp2 directory. If specified, will be passed to # chia plots create as -2. Only one tmp2 directory is supported. # tmp2: /mnt/tmp/a # One or more directories; the scheduler will use all of them. # These again are presumed to be on independent physical devices, # so writes (plot jobs) and reads (archivals) can be scheduled # to minimize IO contention. dst: - /plots # Archival configuration. Optional; if you do not wish to run the # archiving operation, comment this section out. # # Currently archival depends on an rsync daemon running on the remote # host. # The archival also uses ssh to connect to the remote host and check # for available directories. Set up ssh keys on the remote host to # allow public key login from rsyncd_user. # Complete example: https://github.com/ericaltendorf/plotman/wiki/Archiving #archive: #rsyncd_module: plots # Define this in remote rsyncd.conf. #rsyncd_path: /plots # This is used via ssh. Should match path # defined in the module referenced above. #rsyncd_bwlimit: 80000 # Bandwidth limit in KB/s #rsyncd_host: myfarmer #rsyncd_user: chia # Optional index. If omitted or set to 0, plotman will archive # to the first archive dir with free space. If specified, # plotman will skip forward up to 'index' drives (if they exist). # This can be useful to reduce io contention on a drive on the # archive host if you have multiple plotters (simultaneous io # can still happen at the time a drive fills up.) E.g., if you # have four plotters, you could set this to 0, 1, 2, and 3, on # the 4 machines, or 0, 1, 0, 1. # index: 0 # Plotting scheduling parameters scheduling: # Run a job on a particular temp dir only if the number of existing jobs # before [tmpdir_stagger_phase_major : tmpdir_stagger_phase_minor] # is less than tmpdir_stagger_phase_limit. # Phase major corresponds to the plot phase, phase minor corresponds to # the table or table pair in sequence, phase limit corresponds to # the number of plots allowed before [phase major : phase minor]. # e.g, with default settings, a new plot will start only when your plot # reaches phase [2 : 1] on your temp drive. This setting takes precidence # over global_stagger_m tmpdir_stagger_phase_major: 2 tmpdir_stagger_phase_minor: 1 # Optional: default is 1 tmpdir_stagger_phase_limit: 1 # Don't run more than this many jobs at a time on a single temp dir. tmpdir_max_jobs: 4 # Don't run more than this many jobs at a time in total. global_max_jobs: 4 # Don't run any jobs (across all temp dirs) more often than this, in minutes. global_stagger_m: 30 # How often the daemon wakes to consider starting a new plot job, in seconds. polling_time_s: 20 # Plotting parameters. These are pass-through parameters to chia plots create. # See documentation at # https://github.com/Chia-Network/chia-blockchain/wiki/CLI-Commands-Reference#create plotting: k: 32 e: False # Use -e plotting option n_threads: 4 # Threads per job n_buckets: 128 # Number of buckets to split data into job_buffer: 4400 # Per job memory # If specified, pass through to the -f and -p options. See CLI reference. # farmer_pk: ... # pool_pk: ... ```

When starting via plotman plot or plotman interactive, it continually creates plots every 20 seconds (when I had it set to a different dir):

(venv) [plotter@chia-plotter ~/chia-blockchain]$ plotman plot
...starting plot loop
...sleeping 20 s: (True, 'Starting plot job: chia plots create -k 32 -r 4 -u 128 -b 4400 -t /chiatmp -d /plots ; logging to /home/plotter/.chia/logs/2021-05-20T22_38_18.205115-07_00.log')
...sleeping 20 s: (True, 'Starting plot job: chia plots create -k 32 -r 4 -u 128 -b 4400 -t /chiatmp -d /plots ; logging to /home/plotter/.chia/logs/2021-05-20T22_38_38.213755-07_00.log')
...sleeping 20 s: (True, 'Starting plot job: chia plots create -k 32 -r 4 -u 128 -b 4400 -t /chiatmp -d /plots ; logging to /home/plotter/.chia/logs/2021-05-20T22_38_58.508263-07_00.log')
...sleeping 20 s: (True, 'Starting plot job: chia plots create -k 32 -r 4 -u 128 -b 4400 -t /chiatmp -d /plots ; logging to /home/plotter/.chia/logs/2021-05-20T22_39_18.546730-07_00.log')
...sleeping 20 s: (True, 'Starting plot job: chia plots create -k 32 -r 4 -u 128 -b 4400 -t /chiatmp -d /plots ; logging to /home/plotter/.chia/logs/2021-05-20T22_39_38.558798-07_00.log')
...sleeping 20 s: (True, 'Starting plot job: chia plots create -k 32 -r 4 -u 128 -b 4400 -t /chiatmp -d /plots ; logging to /home/plotter/.chia/logs/2021-05-20T22_39_58.570581-07_00.log')
^[[6~^[[B^[[B...sleeping 20 s: (True, 'Starting plot job: chia plots create -k 32 -r 4 -u 128 -b 4400 -t /chiatmp -d /plots ; logging to /home/plotter/.chia/logs/2021-05-20T22_40_18.584677-07_00.log')
^CTraceback (most recent call last):
  File "/usr/home/plotter/chia-blockchain/venv/bin/plotman", line 8, in <module>
    sys.exit(main())
  File "/usr/home/plotter/chia-blockchain/venv/lib/python3.7/site-packages/plotman/plotman.py", line 151, in main
    time.sleep(cfg.scheduling.polling_time_s)
KeyboardInterrupt

Here's a ps output:

(venv) [plotter@chia-plotter ~/chia-blockchain]$ ps aux | grep plots                                                                                                                                                                           
plotter  9580  75.7  0.2 278244 254376  -  INsJ 22:38    3:50.19 /usr/home/plotter/chia-blockchain/venv/bin/python /usr/home/plotter/chia-blockchain/venv/bin/chia plots create -k 32 -r 4 -u 128 -b 4400 -t /chiatmp -d /plots (python3.7)    
plotter 10086  73.8  0.2 278212 254432  -  INsJ 22:39    1:48.63 /usr/home/plotter/chia-blockchain/venv/bin/python /usr/home/plotter/chia-blockchain/venv/bin/chia plots create -k 32 -r 4 -u 128 -b 4400 -t /chiatmp -d /plots (python3.7)    
plotter  9778  72.5  0.2 278212 254416  -  INsJ 22:38    3:00.57 /usr/home/plotter/chia-blockchain/venv/bin/python /usr/home/plotter/chia-blockchain/venv/bin/chia plots create -k 32 -r 4 -u 128 -b 4400 -t /chiatmp -d /plots (python3.7)    
plotter  9982  72.0  0.2 278212 254420  -  INsJ 22:39    2:12.05 /usr/home/plotter/chia-blockchain/venv/bin/python /usr/home/plotter/chia-blockchain/venv/bin/chia plots create -k 32 -r 4 -u 128 -b 4400 -t /chiatmp -d /plots (python3.7)    
plotter  9677  70.3  0.2 278212 254436  -  INsJ 22:38    3:25.46 /usr/home/plotter/chia-blockchain/venv/bin/python /usr/home/plotter/chia-blockchain/venv/bin/chia plots create -k 32 -r 4 -u 128 -b 4400 -t /chiatmp -d /plots (python3.7)    
plotter  9878  69.1  0.2 278244 254424  -  INsJ 22:39    2:36.54 /usr/home/plotter/chia-blockchain/venv/bin/python /usr/home/plotter/chia-blockchain/venv/bin/chia plots create -k 32 -r 4 -u 128 -b 4400 -t /chiatmp -d /plots (python3.7)    
plotter 10198  68.5  0.2 278024 254232  -  INsJ 22:40    1:24.20 /usr/home/plotter/chia-blockchain/venv/bin/python /usr/home/plotter/chia-blockchain/venv/bin/chia plots create -k 32 -r 4 -u 128 -b 4400 -t /chiatmp -d /plots (python3.7)    
plotter 10346  62.1  0.2 278200 254428  -  INsJ 22:41    0:21.07 /usr/home/plotter/chia-blockchain/venv/bin/python /usr/home/plotter/chia-blockchain/venv/bin/chia plots create -k 32 -r 4 -u 128 -b 4400 -t /chiatmp -d /plots (python3.7)    
plotter 10460  46.4  0.2 278244 254424  -  SNsJ 22:41    0:06.14 /usr/home/plotter/chia-blockchain/venv/bin/python /usr/home/plotter/chia-blockchain/venv/bin/chia plots create -k 32 -r 4 -u 128 -b 4400 -t /chiatmp -d /plots (python3.7)    
plotter 10544   0.0  0.0  11296   2608  1  S+J  22:41    0:00.00 grep plots      

I see the logs being created, and the plot is being farmed just fine, but nothing shows up in plotman status or in interactive, and plot creation keeps happening forever.

I poked around with psutil in a python shell, and from what I saw, it should work just fine with parse_chia_plots_create_command_line;

>>> for proc in psutil.process_iter(['pid', 'cmdline']):
...     print( proc.cmdline() )
... 
...skipped a bunch of unrelated things ...
['/usr/home/plotter/chia-blockchain/venv/bin/python', '/usr/home/plotter/chia-blockchain/venv/bin/chia', 'plots', 'create', '-k', '32', '-r', '4', '-u', '128', '-b', '4400', '-t', '/chiatmp', '-d', '/plots']
['/usr/home/plotter/chia-blockchain/venv/bin/python', '/usr/home/plotter/chia-blockchain/venv/bin/chia', 'plots', 'create', '-k', '32', '-r', '4', '-u', '128', '-b', '4400', '-t', '/chiatmp', '-d', '/plots']
['python3.7']
>>> 

I'm not sure what else to check, though. Let me know if I can provide any more logs, debug output, etc!

Edit; I couldn't help myself and did some print() debugging, and found out self.proc.cwd() is failing when creating a new Job object...

>>> p = psutil.Process(pid=14146)
>>> p
psutil.Process(pid=14146, name='python3.7', status='sleeping', started='23:22:30')
>>> p.cwd()
Traceback (most recent call last):
  File "/usr/home/plotter/chia-blockchain/venv/lib/python3.7/site-packages/psutil/_psbsd.py", line 550, in wrapper
    return fun(self, *args, **kwargs)
  File "/usr/home/plotter/chia-blockchain/venv/lib/python3.7/site-packages/psutil/_psbsd.py", line 832, in cwd
    return cext.proc_cwd(self.pid) or None
PermissionError: [Errno 1] Operation not permitted (originated from kinfo_getfile())

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/home/plotter/chia-blockchain/venv/lib/python3.7/site-packages/psutil/__init__.py", line 715, in cwd
    return self._proc.cwd()
  File "/usr/home/plotter/chia-blockchain/venv/lib/python3.7/site-packages/psutil/_psbsd.py", line 557, in wrapper
    raise AccessDenied(self.pid, self._name)
psutil.AccessDenied: psutil.AccessDenied (pid=14146, name='python3.7')

However, if I do the same thing as root, it works... so there must be something stopping that..

[slowbro]

Okay, I fixed it, for FreeBSD. It turns out that you need to twist a sysctl knob in order to have non-privileged users get a process' CWD (as well as some other info). As root, run sysctl security.bsd.unprivileged_proc_debug=1 and then also slap 'security.bsd.unprivileged_proc_debug=1' in your /etc/sysctl.conf - and it works now!!

 plot id    k        tmp      dst   wall   phase    tmp     pid   stat    mem   user    sys   io
4518e539   32   /chiatmp   /plots   0:40     1:3   114G   14503    SLP   5.8G   1:17   0:02    -
4144c410   32   /chiatmp   /plots   0:44     1:3   124G   14146    SLP   5.8G   1:25   0:02    -

Total jobs: 2
Jobs in /chiatmp: 2