Allow executing without patching cv

[totten]

Overview

Add the pop console command directly into pop.git

Background/Motivation

@michaelmcandrew I needed some sample data recently, so I wondered if I could try pop again.

In the previous work, it required adding everything to cv (https://github.com/civicrm/cv/pull/3). The discussion was a bit long, but the final problem (where the PR stalled) was the raw size of the dependencies -- Faker needs a lot of data, and embedding it incv.phar made it too heavy. So we needed a way to run it without embedding the whole thing in cv.phar.

With this PR, pop is a standalone command. To connect to CiviCRM, it calls cv pipe (aka Civi::pipe(); Pipe Reference). The advantage of this arrangement:

• Near-native performance -- The pipe only boots CiviCRM one time. When you're sending 1000x API calls, it doesn't have the same overhead as HTTP/REST or cv api3 Entity.action.
• Flexible dependencies -- You can freely use any PHP packages without dependency conflicts.
  • For example, "Symfony Console" is great but has dephell when you navigate between D7/D8/D9/D10. In this setup, "pop" can pick whatever version of "Symfony Console" it prefers. (It can also use other libraries like clippy.)

[totten]

So I've gotten a lot further with using pop than I did in prior years. My goal today was to load large dummy data-set. In particular, I ran these:

• examples/basic-10k.yml (10,000 individuals) worked. Runtime was around 10 min.
• examples/basic-100k.yml (100,000 individuals) eventually worked (notwithstanding some snags -- like overfilling the mysql-partition). Runtime was around 2 hours.
• (Running on Ubuntu with i3-12100 and mysql-ramdisk.)

After running these, I just wanted to braindump some observations/speculations:

• Loading example-data through the API means you can use the API's metadata to help choose random values. It is pretty cool how little configuration was required. Basically just a little YAML (e.g. Individual: 10000 and GroupContact: 20000).
• Loading example-data through the API isn't fast. For sporadic dev-test purposes, I can live with 10 min. But 2h isn't realistic.
• Loading a SQL snapshot is way, way faster. (For the 100k dataset, civibuild restore takes 40s -- 300x faster.)
• I maintain that the pipe is faster than REST or CLI, but I did find a rough spot:
  • Specifically: pop calls Contact.get to retrieve all CIDs. For 100k contacts, that response is quite large and requires a large buffer. It failed with 100mb(!) buffer. So I bumped up to 200mb.
• It should be faster if we introduce parallel processing -- e.g. throw the work into a queue and then run coworker -d maxConcurrentWorkers=8. (Best-case-scenario: 2h becomes 15m)
  • As a bonus, it might be a nice demo of queuing.
  • But this would still be slower than SQL (15m vs 40s).
• It should be even faster if one generates a SQL file (pop basic-100k.yml > basic-100k.mysql) instead of invoking APIs.
  • As a bonus, you can re-use that SQL file.
  • But the metadata-based generation of random-data may not work as nicely.
• I do not know if upgrading from APIv3 to APIv4 would help performance. (v4 is faster for complex reads. For writing basic records, it would be similar?)
• When pop generates a record like Participant or GroupContact, it needs to pick some random FK IDs (per Populator and EntityStore). It may be worth changing the approach:
  • The current technique reads all IDs and then picks a random one. With 100k contacts, this works but requires high resources. The overhead is probably prohibitive for 1m contacts.
  • An alternative strategy: as you insert records, track the first_contact_id and last_contact_id. Choose a random FK as contact_id = rand(first_contact_id, last_contact_id). The strategy should be OK for newly-generated example-records (which form a contiguous batch).

[michaelmcandrew]

@totten merging this. glad it was useful. haven't used it for years. feel free to take ownership if you like :)