search

freelawproject / courtlistener

A fully-searchable and accessible archive of court data including growing repositories of opinions, oral arguments, judges, judicial financial records, and federal filings.
https://www.courtlistener.com
Other
531 stars 147 forks source link

Build a citator for cases #3661

Open mlissner opened 7 months ago

mlissner commented 7 months ago

Just got another request from a client to build a sepharizing tool that can flag or score cases as good or bad law. We've talked about this a lot, and considered approaches like regexes for easy stuff + ML for hard. I think AI probably would be a useful tool here too.

Some thing we'll need to do to get this started:

  1. We need a target. What would we want this to look like and what features should it have?
  2. We need an ontology for our flags. Which flags make sense?
  3. We need a name for the feature. Prior art includes "Sepharizer" and FastCase's "Bad Law Bot."

For now this is something of a placeholder, but I certainly welcome discussion and ideas. It'd be a heck of a project for an intrepid developer to pick up and run with, but I suspect the way to do it correctly is to have a dedicated engineer on it for many months.

I'd expect this to be a duplicate, since we've talked about it so many times, but I can't find any issue for this. Hrm.

anseljh commented 1 month ago

Here's my notes dump! I may come back to add more to this later.

Functions of a Citator (what it does)

Attributes of a Citator (what it is/has)

Existing and Previous Citators

Quasi-citators

News items about citators

Datasets helpful for citators

Potential differentiating features of a new citator

Example Module Ideas

mlissner commented 1 month ago

This is great, Ansel! Very, very helpful and lots to go over. Really emphasizes how well-developed this area already is.

It'd be great to think about other ways to make something even better than the past. For example, Paxton mentions that their citator can group similar cases together. That's an interesting idea. I'm not sure how it works, but I could see a citator that:

There are probably a dozen similar ideas we should explore as we're launching this.

anseljh commented 1 month ago

Important to fix https://github.com/freelawproject/courtlistener/issues/4290 for this.

mlissner commented 4 weeks ago

Features from SmartCite (CaseText):

mlissner commented 4 weeks ago

Two thoughts from @s-taube discussion today:

  1. It should have a feedback button that users can click if they don't like a flag.
  2. Being able to adjust how conservatively flags are applied as a user could be interesting/useful. This implies that red/green are backed by values between zero and one, instead of being simply binary, but imagine wanting to be really careful and saying that green flags should only apply to the most safe cases, say.
anseljh commented 3 weeks ago
  1. Being able to adjust how conservatively flags are applied as a user could be interesting/useful. This implies that red/green are backed by values between zero and one, instead of being simply binary, but imagine wanting to be really careful and saying that green flags should only apply to the most safe cases, say.

I like this idea, but I think it would require substantial user testing among lawyers and other professionals who use citators to figure out how to communicate it effectively. Thinking probabilistically about a citator (or really anything?) would be a big mindset shift.

Counterpoint: Citator users don't understand the nuances of market-leading products all that well either. Rebecca is working on a paper that touches on this. Ask her about it!

Counter-counterpoint: The bar is higher for new entrants.

At Syntexys, we used a little linear gauge to display confidence of contract clause-type predictions, which worked okay. In this context, though, Lexis and Westlaw each do something similar already to display depth of treatment, so citator users are used to that type of widget meaning something other than probability/confidence scores.

Lexis/Shepard's: image https://perma.cc/C6PS-PCBF

Westlaw/KeyCite: image https://perma.cc/B6JF-2NLH

legaltextai commented 2 weeks ago

This is a super useful summary for a new entrant like myself. Thank you @anseljh.

Quick two thoughts:

1) With all the mentioned available datasets where we may have a text-classification pair like: "Whatever may have been the extent of psychological knowledge at the time of Plessy v. Ferguson, this finding is amply supported by modern authority. Any language in Plessy v. Ferguson contrary to this finding is rejected." = negative treatment

Do we want to fine-tune a language model with good reasoning skills on these datasets? The idea is to train the model to be able to recognize the type of treatment based on the input text.

2) Do we want to start with training the model to be able to extract all cases? In my early attempts, even GPT-4 and Claude 3.5 would not extract all cited cases, probably around 70-80% of all cited cases in a given opinion.

Briefs from here could be a good starting point. The list of extracted cases by the model will be compared against the ones mentioned in Table of Authorities.