4682 Updated cl_send_alerts to support V2 webhooks while maintaining compatibility with V1 webhooks

[albertisfu]

As planned in #4682, this PR updates the cl_send_alerts command to support V2 of the Opinion Search Alerts Webhooks.

The following changes have been applied:

• All Opinion Search Alerts emails now use the new email format, which displays nested opinions within the Opinion Cluster. This is achieved doing an ES parent-child query all the time as main query.
• If the user has a V1 Webhook enabled, an additional query is performed to retrieve plain results for the alert. This is executed via the MultiSearch API, allowing both the parent-child query and the plain query to be requested in a single request.
  • Nested results from the main query are used to render the email and V2 Webhook.
  • Plain results are used to send V1 Webhooks.
  • It's important to note that V1 Webhooks use a collapse query to group Opinions belonging to the same cluster. This aligns with V3 of the Opinions Search API running on ES, meaning that the same ES V3 API backwards incompatible changes for opinions apply to V1 Webhooks.
• These changes will take effect once we switch to ES for Opinions. Until that happens, we will continue sending only V1 Webhooks using Solr and the regular plain email, even if users have enabled a V2 Webhook endpoint for Search Alerts.
• Refactored the code used to merge child documents in nested queries to eliminate duplicate and redundant code across the codebase.
• Updated the Opinion Search Alerts tests accordingly.

Below are examples of the updated Opinion Search Alert Email and Webhooks:

*****************
CourtListener.com
*****************

-------------------------------------------------------
We have news regarding your alerts at CourtListener.com
-------------------------------------------------------

Your daily opinion alert -- Test Alert O Disabled -- had 1 hit:
-------------------------------------------------------
View Full Results / Edit this Alert: https://www.courtlistener.com/?type=o&stat_Unpublished=on&order_by=score+desc&filed_after=2024-11-14&edit_alert=23
Disable this Alert (one click): https://www.courtlistener.com/alert/disable/vAFtMQ10lUvXLwCqCqhxzTm4EBcWDDsTJFHinvyS/

1. California vs Lorem (Bankr. C.D. Cal. 2024)

Lead Opinion
   ...Lorem dolor sit amet, consectetur adipiscing elit hearing....

Combined Opinion
   ...Lorem dolor california sit amet, consectetur adipiscing elit....
    - Download original from the court: https://ca.flcourts.gov/
    - Download the original from our backup: https://storage.courtlistener.com/test/search/opinion_html.html

~~~~~
 - View this item on our site: https://www.courtlistener.com/opinion/11/california-vs-lorem/

************************
This alert brought to you by the 501(c)(3) non-profit Free Law Project
 - Blog: https://free.law
 - BlueSky: https://bsky.app/profile/free.law
 - X: https://x.com/freelawproject
 - Donate: https://donate.free.law/forms/supportflp
 - Become a Member: https://donate.free.law/forms/membership

Please donate to support our work.

Webhook V1

{
   "payload":{
      "alert":{
         "id":24,
         "name":"Test Alert 2 O Enabled",
         "rate":"dly",
         "user":14,
         "query":"q=California hearing&type=o&stat_Unpublished=on",
         "alert_type":"o",
         "secret_key":"LmdVe5PURP0mehQSmDdmdMn6OeaLBafHEdMFpMmB",
         "date_created":"2024-11-18T16:22:15.756423-08:00",
         "date_last_hit":"2024-11-14T16:22:15.765337-08:00",
         "date_modified":"2024-11-14T16:22:15.765337-08:00"
      },
      "results":[
         {
            "id":13,
            "type":"020lead",
            "cites":"None",
            "court":"Superior court of the Dirty Dishes",
            "judge":"Lorem Judge",
            "status":"Non-Precedential",
            "scdb_id":"",
            "snippet":"Lorem dolor sit amet, consectetur adipiscing elit <strong>hearing</strong>.",
            "attorney":"Attorney General of North Carolina",
            "caseName":"California vs Lorem",
            "citation":[
               "10 Yeates 4",
               "10 Neutral 4",
               "33 state 1"
            ],
            "court_id":"canb",
            "author_id":16,
            "citeCount":1,
            "dateFiled":"2024-11-14T00:00:00-08:00",
            "docket_id":13,
            "lexisCite":"10 Yeates 4",
            "panel_ids":[
               15
            ],
            "timestamp":"2024-11-14T16:22:15.765337-08:00",
            "cluster_id":11,
            "dateArgued":"1986-04-23T00:00:00-08:00",
            "local_path":"None",
            "per_curiam":false,
            "suitNature":"",
            "court_exact":"canb",
            "neutralCite":"10 Neutral 4",
            "sibling_ids":[
               13,
               14
            ],
            "absolute_url":"/opinion/11/california-vs-lorem/",
            "dateReargued":"2024-11-14T00:00:00-08:00",
            "date_created":"2024-11-14T16:22:15.765337-08:00",
            "docketNumber":"2:13-bk-70798",
            "download_url":"None",
            "joined_by_ids":[
               15
            ],
            "dateReargumentDenied":"2024-11-14T00:00:00-08:00",
            "court_citation_string":"Bankr. C.D. Cal."
         }
      ]
   },
   "webhook":{
      "version":1,
      "event_type":2,
      "date_created":"2024-11-19T00:22:15.753650+00:00",
      "deprecation_date":"2024-11-18T00:00:00+00:00"
   }
}

Webhook V2

{
   "payload":{
      "alert":{
         "id":17,
         "name":"Test Alert O",
         "rate":"dly",
         "user":11,
         "query":"q=California&type=o&stat_Unpublished=on",
         "alert_type":"o",
         "secret_key":"ivjg1J15qU8jgE70QHuI4XQkUciVuF7BBcmjAbDE",
         "date_created":"2024-11-18T16:22:15.707344-08:00",
         "date_last_hit":"2024-11-14T16:22:15.765337-08:00",
         "date_modified":"2024-11-14T16:22:15.765337-08:00"
      },
      "results":[
         {
            "meta":{
               "timestamp":"2024-11-15T00:22:15.765337Z",
               "date_created":"2024-11-15T00:22:15.765337Z"
            },
            "court":"Superior court of the Dirty Dishes",
            "judge":"Lorem Judge",
            "source":"CR",
            "status":"Unpublished",
            "posture":"",
            "scdb_id":"",
            "attorney":"Attorney General of North Carolina",
            "caseName":"<strong>California</strong> vs Lorem",
            "citation":[
               "10 Yeates 4",
               "10 Neutral 4",
               "33 state 1"
            ],
            "court_id":"canb",
            "opinions":[
               {
                  "id":14,
                  "meta":{
                     "timestamp":"2024-11-15T00:22:15.765337Z",
                     "date_created":"2024-11-15T00:22:15.765337Z"
                  },
                  "sha1":"d7bcc865b883abd70d74d9af7578d256ae62a973",
                  "type":"combined-opinion",
                  "cites":[
                     13
                  ],
                  "snippet":"Lorem dolor <strong>california</strong> sit amet, consectetur adipiscing elit.",
                  "author_id":17,
                  "local_path":"test/search/opinion_html.html",
                  "per_curiam":false,
                  "download_url":"https://ca.flcourts.gov/",
                  "joined_by_ids":[

                  ]
               },
               {
                  "id":13,
                  "meta":{
                     "timestamp":"2024-11-15T00:22:15.765337Z",
                     "date_created":"2024-11-15T00:22:15.765337Z"
                  },
                  "sha1":"58ccc5a2c60c28bf04fe8efc78547270c360f208",
                  "type":"lead-opinion",
                  "cites":[

                  ],
                  "snippet":"Lorem dolor sit amet, consectetur adipiscing elit hearing.",
                  "author_id":16,
                  "local_path":"None",
                  "per_curiam":false,
                  "download_url":"None",
                  "joined_by_ids":[
                     15
                  ]
               }
            ],
            "syllabus":"",
            "citeCount":1,
            "dateFiled":"2024-11-14",
            "docket_id":13,
            "lexisCite":"10 Yeates 4",
            "panel_ids":[
               15
            ],
            "cluster_id":11,
            "dateArgued":"1986-04-23",
            "suitNature":"",
            "neutralCite":"10 Neutral 4",
            "panel_names":[
               "Kristine Pena Mendez Jr."
            ],
            "sibling_ids":[
               13,
               14
            ],
            "absolute_url":"/opinion/11/california-vs-lorem/",
            "caseNameFull":"Todd Ballard, Julie Smith, Jeanne Peters, Ashley Jordan, and Andrew Thompson v. Johnson LLC, Walker-Bernard, Jones PLC, Bailey Group, and Lopez-Gonzales",
            "dateReargued":"2024-11-14",
            "docketNumber":"2:13-bk-70798",
            "procedural_history":"",
            "dateReargumentDenied":"2024-11-14",
            "court_citation_string":"Bankr. C.D. Cal.",
            "non_participating_judge_ids":[

            ]
         }
      ]
   },
   "webhook":{
      "version":2,
      "event_type":2,
      "date_created":"2024-11-19T00:22:15.511388+00:00",
      "deprecation_date":"None"
   }
}

Let me know what do you think.

[mlissner]

Process notes (@flooie and @s-taube, you might want to perk your ears, so we're all on the same page):

• I assigned to Eduardo so it shows up in his "By Assignee" tab.
• I set Eduardo as the reviewer, because he is.
• I moved it to the To Do column so it shows up in Eduardo's todo column.
• @ERosendo, when you finish your review, see if you can assign this back to Alberto, so it shows up in his queue, and then back and forth until you're done.

I think this should work, but I'm still experimenting. Definitely curious what we think.