Add synonyms to our search engine for common gov't agencies and acts

[mlissner]

It'd be nice if a search for FBI brought back results for Federal Bureau of Investigations, but it's a bit weird if a search for HSC brings back results for the less-popular Homeland Security Council. Some agencies are just more popular than others and our synonym file should reflect that.

Furthermore, we need a list of all the important acts so we can have things like FOIA and Freedom of Information Act be synonyms, but, probably many fewer people have heard of the APA - The Animal Protection Act.

(That said, I'll insist that CETA—The Courthouse Ethics and Transparency Act—be in the system as an easter egg, since it's the bill we helped pass.)

This is a human task that a volunteer could really help with, by making a simple spreadsheet with this curated info. If multiple people want to work together, that's even better.

There are a number of sources of this information:

• https://www.usa.gov/agency-index/
• https://en.wikipedia.org/wiki/List_of_federal_agencies_in_the_United_States

@joshdata I haven't looked much, but do you have any thoughts on how to find popular acts and their acronyms? This feels like your bread and butter?

[flooie]

@mlissner have you considered just pulling them from our opinions?

[JoshData]

The canonical answer for popular titles of acts is https://uscode.house.gov/popularnames/popularnames.htm;jsessionid=98FCB2CE4C3188CF1B0E496FFD04FA35. I have long dreamed of using this but have never gotten the chance. It's possible this is in the USCode XML but I don't recall. This is probably the best place to start. (Would love to collaborate.)

I've scraped Wikipedia for pages tagged as legislation (to import summaries into GovTrack) and using the page titles could also be interesting. Scraper: https://github.com/govtrack/govtrack.us-web/blob/4deab85373fe75bb1407bbe92197abb079d0b37b/bill/wikipedia.py#L2

[anseljh]

Is there a query to Wikidata that would give you some of this?

[mlissner]

Thanks Josh! What's your interest? Do you need a synonym file too?

We just had a big conversation on our Slack about this. It sort of feels like the best thing to do is to download the opinions bulk data file:

https://storage.courtlistener.com/bulk-data/opinions-2023-05-31.csv.bz2

And to then go through it with a regex to find and tally anything that looks like an acronym. @flooie did some really basic stuff with this regex:

m = re.search(r"(\w+\W+){10}\(["\"]?[A-Z]{3,}["\"]?\)", t, flags=re.MULTILINE)

[mlissner]

Oh, and re this:

Is there a query to Wikidata that would give you some of this?

I meant to say that I have no idea and I don't know anybody that does either! Wikidata often feels like the solution, but I never know anybody that knows how to make use of it.

[JoshData]

Do you need a synonym file too?

Unclear. It's possible that for legislation I already have this via bill titles, but I'm not sure how far back titles are comprehensive, and I don't have anything for statutes before 1973.

Wikidata often feels like the solution, but I never know anybody that knows how to make use of it.

Hahah same.

[flooie]

When I ran the code above - with some slight modifications I was able to claw back

New York City Housing Authority (NYCHA)                                                              NYCHA
Police Pension Fund (PPF)                                                                            PPF
Real Property Tax Law (RPTL)                                                                         RPTL
Taxi and Limousine Commission (TLC)                                                                  TLC
Periodic Payment Settlement Act of 1982 (PPSA)                                                       PPSA

                 ................MANY MORE ITEMS......................

Federal Arbitration Act (FAA)                                                                        FAA
Presentence Investigation Report (PSR)                                                               PSR
Bureau of Prisons (BOP)                                                                              BOP
False Claims Act (FCA)                                                                               FCA
Appellees explain, “the Administrative Office of Pennsylvania Courts (AOPC)                          AOPC
presentence report                                                                                   PSR
Ramses Air Freight & Transport, Inc. (RAFT)                                                          RAFT

This from a short burst of API queries against the plain text. It's not perfect, but I suspect if it was tailored nicely to the federal courts. you could get a nice collection of abbreviations going back as far as you want.

[mlissner]

Josh said:

The canonical answer for popular titles of acts is ....

So, I just took a look at that, and woah. It's not exactly popular stuff in the sense I was hoping. Just a glance there's all kinds of boring bills in there. Seems like it's more of a "Common Names" index than "popular."

With this in mind, I'm going to see if I can make an export that's useful to parse. For our purposes, this will be a "Commonly litigated names" index, and it'll go beyond just acts, but include any acronyms in the past XX years that are commonly in court cases.

[mlissner]

Cool, so I remembered we have a 10% sample of CourtListener's data that we use to benchmark eycite's performance as we tweak it, so I'm using some of @flooie's code to process that. It'll finish soon.

When it does, I'll have a CSV of all the acronyms in a 10% sample of CourtListener, as a CSV. From there, I'll have two steps:

1. Figure out which lines matter

   I'll use common sense and counts for this.

2. Eliminate acronyms that are also regular English words.

   This is needed so that searching for song doesn't bring back results for San Onofre Nuclear Generating Station. Although SONGS is a fine nuclear plant that I used to drive by from time to time, I don't want musicians to get back nuclear results.

Does anybody have a trick for identifying English words from a list?

[mlissner]

Turns out 10% only nets about 1000 acronyms. I'm going to review the regex to see if it's too tight, but while I do that I'm generating a 30% file too.

[flooie]

10% file may not have 10% plain text which is what I built it for.

[mlissner]

I tweaked it. The current code is:

In [54]: with open('ten-percent.csv', 'r') as in_f, open("abbrev2.csv", 'w') as out:
    ...:     csv_data = csv.DictReader(in_f, delimiter=',')
    ...:     csv_out = csv.writer(out)
    ...:     for i, row in enumerate(csv_data):
    ...:         text = (
    ...:                 row["xml_harvard"]
    ...:                 or row["html_lawbox"]
    ...:                 or row["html_columbia"]
    ...:                 or row["html_anon_2020"]
    ...:                 or row["html"]
    ...:                 or row["plain_text"]
    ...:             )
    ...:         m = re.search(r"(\w+\W+){10}\(\"?([A-Z]{3,5})\"?\)", text, flags=re.MULTILINE)
    ...:         if not m:
    ...:             print(f"No match for item {i}")
    ...:             continue
    ...: 
    ...:         words = m.group().split()
    ...:         abbreviation = m.groups(2)[1]
    ...: 
    ...:         matching_now = False
    ...:         capitalized_words = []
    ...:         for word in words:
    ...:             if word[0] == abbreviation[0]:
    ...:                 matching_now = True
    ...:             if matching_now and not word.startswith('('):
    ...:                 capitalized_words.append(word)
    ...:         if capitalized_words:
    ...:             phrase = ' '.join(capitalized_words)
    ...:             print(f"{i}\t{phrase}\t{abbreviation}")
    ...:             csv_out.writerow([row["id"], i, phrase, abbreviation])
    ...:         else:
    ...:             sentence =  " ".join(words[-len(abbreviation)-2:])
    ...:             #print(f"{sentence=}")
    ...: 
    ...:             consec = find_consecutive_words(sentence, abbreviation, True)
    ...:             if not consec:
    ...:                  consec = find_consecutive_words(sentence, abbreviation, False)
    ...:             if consec and len(consec[0]) > len(abbreviation):
    ...:                 csv_out.writerow([row["id"], i, consec[0], abbreviation])
    ...:                 print(f"{i}\t{consec[0]}\t{abbreviation}")
    ...:         out.flush()

[mlissner]

Fact: I've been playing with 0.1% and 0.3% samples of the DB and not realizing it. Brace for more data.

[mlissner]

OK, I generated a 1% file of cases since 1975 with:

root@maintenance:/opt/courtlistener# PGPASSWORD=$DB_PASSWORD psql       --command         "set statement_timeout to 0; COPY (      SELECT        s.id, s.plain_text, s.html, s.html_lawbox, s.html_columbia, s.html_anon_2020, s.xml_harvard       FROM         search_opinion s  TABLESAMPLE BERNOULLI(1) INNER JOIN search_opinioncluster c ON s.cluster_id = c.id WHERE c.date_filed > '1975-01-01'  )    TO STDOUT WITH (FORMAT csv, ENCODING utf8, HEADER)"    --host $DB_HOST         --username $DB_USER     --dbname courtlistener |        bzip2 |         aws s3 cp - s3://com-courtlistener-storage/bulk-data/eyecite/tests/one-percent-1975-2023.csv.bz2 --acl public-read

That's living at: https://storage.courtlistener.com/bulk-data/eyecite/tests/one-percent-1975-2023.csv.bz2

wc -l reports that it's about 7.5M lines, which doesn't make sense at first, except several of the columns contain text fields with new lines. I'm sort of impressed that Python's csv can ingest the same thing that postgresql is outputting, and maybe it's not, but I can live with a bit of sloppiness here.

I'm generating a CSV of acronyms now.

[mlissner]

OK, here's the latest code:

def find_consecutive_words(text, letters, spacing):
    """"""
    if spacing:
        ptn = [fr'\b{x}[\w-]*\b' for x in letters]
        pattern = "\s".join(ptn)
        matches = re.findall(pattern, text, flags=re.IGNORECASE)
    else:
        ptn = [fr'{x}[\w-]*\s?' for x in letters]
        k = "\\b" + "".join(ptn) + "\\b"
        matches = re.findall(k, text, flags=re.IGNORECASE)
    return matches

wrote = 0
with open('one-percent-1975-2023-random-order.csv', 'r') as in_f, open("abbrev-1p-1975-random-order.csv", 'w') as out:
    csv_data = csv.DictReader(in_f, delimiter=',')
    csv_out = csv.writer(out)
    for i, row in enumerate(csv_data):
        if wrote == 25_000:
            break
        text = (
                row["xml_harvard"]
                or row["html_lawbox"]
                or row["html_columbia"]
                or row["html_anon_2020"]
                or row["html"]
                or row["plain_text"]
            )
        m = re.search(r"(\w+\W+){10}\(\"?([A-Z]{3,5})\"?\)", text, flags=re.MULTILINE)
        if not m:
            print(f"No match for item {i}")
            continue

        words = m.group().split()
        abbreviation = m.groups(2)[1]

        matching_now = False
        capitalized_words = []
        for word in words:
            if word[0] == abbreviation[0]:
                matching_now = True
            if matching_now and not word.startswith('('):
                capitalized_words.append(word)
        if capitalized_words:
            phrase = ' '.join(capitalized_words)
            print(f"{i}\t{phrase}\t{abbreviation}")
            csv_out.writerow([row["id"], i, phrase, abbreviation])
            wrote += 1
        else:
            sentence =  " ".join(words[-len(abbreviation)-2:])
            #print(f"{sentence=}")

            consec = find_consecutive_words(sentence, abbreviation, True)
            if not consec:
                 consec = find_consecutive_words(sentence, abbreviation, False)
            if consec and len(consec[0]) > len(abbreviation):
                csv_out.writerow([row["id"], i, consec[0], abbreviation])
                wrote += 1
                print(f"{i}\t{consec[0]}\t{abbreviation}")
        out.flush()

I used that to process a randomly ordered random sample of one percent of cases from 1975 to today.

The resulting CSV has about 8k lines:

abbrev-1p-1975-random-order.csv

Still hoping somebody knows how to identify English words vs acronyms.

[flooie]

What do you mean English words vs acronyms?

[flooie]

I wonder if limiting it to 3 to 5 characters was both too long and too short

[mlissner]

If an acronym happens to be an English word, I want to identify that so that we don't include it in our synonym file. Imagine the made-up "Judicial Urgency Dashboard Generation Expedience Act (JUDGE)". We don't want that act to come up every time somebody searches for judge.

[flooie]

ah...

[mlissner]

Sure, exactly.

[flooie]

I wish we could identify that it could be an acronym or abbreviation and put a note at the top of search or something - like - were you looking for Mississippi Alternative Path?

[mlissner]

That'd take some custom Python stuff, but I don't think it's terribly important to get this perfect.

[flooie]

then - why not just run the found terms against a simple dictionary and drop out anything that has a matching word.

[mlissner]

Right, well, pluralization and things like that are tricky, and I don't have a dictionary, but yeah, that's probably where we're headed. Or maybe I'll do it by eye.

[flooie]

I think this is a great tool

https://pypi.org/project/wordfreq/

they pulled lots of data to just identify word frequency, which may not always work to identify common acronyms but should be good at approving most.

[mlissner]

I'll take a look. In the meantime, here's the top ten:

Phrase	Abbrev	Count
administrative law judge	ALJ	152
Board of Immigration Appeal	BIA	145
Bureau of Indian Affairs	BIA	145
Employee Retirement Income Security Act	ERISA	90
Internal Revenue Service	IRS	79
presentence report	PSR	67
Department of Homeland Security	DHS	65
Department of Human Services	DHS	65
Post Conviction Relief Act	PCRA	65
Equal Employment Opportunity Commission	EEOC	64

It's not perfect; you can the Bureau of Indian Affairs and the Board of Immigration Appeal worked together, but it's good enough.

I'll post a final CSV in a few minutes.

[mlissner]

OK, here's the list of the top abbreviations in legal documents since 1975, in Solr synonym format:

american arbitration association => aaa
american bar association, aba
american broadcasting company, abc
armed career criminal act, acca
arkansas crime information center, acic
americans with disabilities act, ada
age discrimination in employment act, adea
attention deficit hyperactivity disorder, adhd
antiterrorism and effective death penalty act, aedpa
aid to families with dependent children, afdc
air force office of special investigations, afosi
american institute of architects, aia
acquired immune deficiency syndrome, aids
administrative law court, alc
administrative law judge, alj
american national standards institute, ansi
agency of original jurisdiction, aoj
administrative procedure act, administrative procedures act, apa
albuquerque police department, apd
armed services board of contract appeals, asbca
alcohol tobacco and firearms, atf
all terrain vehicle, atv
assistant united states attorney, ausa
blood alcohol concentration, bac
bankruptcy appellate panel, bap
bureau of criminal apprehension, bca
bureau of criminal identification, bci
board of immigration appeal, bureau of indian affairs, bia
bureau of land management, blm
bureau of prisons, bop
benefits review board, brb
child advocacy center, children advocacy center, cac
coronavirus aid relief and economic security, cares
convention against torture => cat
collective bargaining agreement, cba
continuing criminal enterprise, cce
california department of corrections and rehabilitation, cdcr
controlled dangerous substance, cds
chief executive officer, code enforcement officer, ceo
california environmental quality act, ceqa
commercial general liability, cgl
california highway patrol, chp
criminal investigation division, cid
court improvement program, cip
certificate of appealability, coa
consolidated omnibus budget reconciliation act, cobra
consumer protection act, cpa
chicago police department, cpd
child protective services, children protective services, cps
california rehabilitation center, crc
citizens for responsibility and ethics in washington => crew
controlled substances act, csa
child sexual abuse accommodation syndrome, csaas
child support recovery act, civil service reform act, civil service retirement act, csra
conditional use permit => cup
connecticut unfair trade practices act, cutpa
clean water act, cwa
children and youth services, cys
department of children and families, dcf
department of children and family services, division of child and family services, dcfs
department of child services, dcs
drug enforcement administration, dea
department of environmental conservation, dec
department of environmental protection, dep
department of environmental quality, deq
department of environmental resources, der
department of family and children services, dfacs
division of family services, dfs
division of housing and community renewal, dhcr
department of health and human services, dhhs
department of human resources, dhr
department of homeland security, department of human services, dhs
disability insurance benefits, dib
department of juvenile justice, djj
department of local government finance, dlgf
department of motor vehicles, dmv
deoxyribonucleic acid, dna
department of natural resources, dnr
department of correction, doc
department of correctional services, docs
department of defense, dod
department of education, doe
department of health, doh
department of labor, dol
department of revenue, dor
department of transportation => dot
department of transportation and development, dotd
department of public safety, dps
department of public welfare, dpw
development review board, drb
department of social and health services, dshs
department of social services, dss
deceptive trade practices act, deceptive trade practices consumer protection act, dtpa
driving under the influence, dui
domestic violence restraining order, dvro
driving while intoxicated, dwi
division of youth and family services, dyfs
equal access to justice act, eaja
equal employment opportunity commission, eeoc
environmental impact statement, eis
emergency medical services, ems
environmental protection agency, epa
employee retirement income security act, erisa
federal arbitration act, federal aviation administration, faa
first amended complaint, fac
federal bureau of investigation, fbi
false claims act, federal communications act, fca
federal communications commission, fcc
fair credit reporting act, fcra
food and drug administration, fda
fair debt collection practices act, fdcpa
federal deposit insurance corporation, fdic
federal energy administration, fea
federal election campaign act, feca
fair employment and housing act, feha
federal employers liability act, fela
federal emergency management agency, fema
federal energy regulatory commission, ferc
fair housing act, fha
federal insurance contribution act, fica
financial industry regulatory authority, finra
fair labor standards act, flsa
family and medical leave act, fmla
freedom of information act, foia
freedom of information law => foil
federal rules of civil procedure, frcp
fair sentencing act, fsa
foreign sovereign immunities act, fsia
federal savings and loan insurance corporation, fslic
federal trade commission, ftc
federal tort claims act, ftca
guardian ad litem, gal
georgia bureau of investigation, gbi
government employees insurance company, geico
global positioning system, gps
general services administration, gsa
horizontal gaze nystagmus, hgn
health and human services, hhs
health insurance portability and accountability act, hipaa
human immunodeficiency virus, hiv
health maintenance organization, hmo
housing preservation and development, hpd
hawaii revised statutes, hrs
harmonized tariff schedule of the united states, htsus
housing and urban development, hud
interstate agreement on detainers, iad
initial appeal file, iaf
interstate commerce commission, icc
immigration and customs enforcement, ice
intensive care unit, icu
indian child welfare act, icwa
individuals with disabilities education act => idea
illinois department of corrections, idoc
individualized education plan, iep
in forma pauperis, ifp
independent medical examination, ime
immigration and nationality act, insurance company of north america, ina
immigration and naturalization service, ins
illinois pattern jury instruction, ipi
individual retirement account, ira
internal revenue service, irs
international trade administration, ita
intrauterine device, iud
judge of compensation claims, jcc
kentucky bar association, kba
kansas offender registration act, kansas open records act, kora
kentucky revised statutes, krs
kansas sentencing guidelines act, ksga
longshore and harbor workers compensation act, lhwca
louisiana insurance guaranty association, liga
life insurance company of north america, lina
limited liability company, llc
labor management relations act, lmra
massachusetts bay transportation authority, mbta
metropolitan correctional center, mcc
mississippi department of corrections, mdoc
mortgage electronic registration systems, mers
memorandum of understanding, mou
municipalities planning code, mpc
metropolitan police department, mpd
magnetic resonance imaging, mri
marital settlement agreement, msa
merit systems protection board, mspb
mississippi tort claims act, mtca
mandatory victims restitution act, mvra
national association of securities dealers, nasd
national crime information center, ncic
national environmental policy act, nepa
no early release act, nera
national highway traffic safety administration, nhtsa
national labor relations act, nlra
national labor relations board, nlrb
notice of appeal, noa
notice of disagreement, nod
new york city police department, new york police department, nypd
office of administrative hearings, oah
oklahoma bar association, oba
ohio bureau of employment services, obes
office of consumer advocate, oca
office of disciplinary counsel, odc
ohio department of transportation, odot
ohio department of rehabilitation and correction, odrc
office of lawyer regulation, olr
open meetings act, oma
office of mental health, omh
omaha police department, opd
office of personnel management, opm
occupational safety and health act, occupational safety and health administration, osha
office of workers compensation, owc
office of workers compensation programs, owcp
operating while intoxicated, owi
public access to court electronic records, pacer
post conviction relief, pcr
post conviction relief act, pcra
public employee relations act, pera
public employment relations commission, perc
public employees retirement system, pers
pennsylvania human relations act, phra
personal injury protection, pip
prison litigation reform act, plra
public records act, pra
personal restraint petition, prp
public service commission, psc
presentence investigation, psi
presentence report, psr
political subdivisions tort claims act, pstca
permanent total disability, ptd
patent and trademark office, pto
post traumatic stress disorder, posttraumatic stress disorder, ptsd
public utilities commission, puc
planned unit development, pud
public utility regulatory act, pura
qualified domestic relations order, qdro
rules of the united states court of federal claims, rcfc
request for proposal, rfp
religious freedom restoration act, rfra
rules governing disciplinary proceedings, rgdp
racketeer influenced and corrupt organization act, rico
revised judicature act, rja
railway labor act, rla
rules on lawyers professional responsibility, rlpr
resolution trust corporation, rtc
second amended complaint, sac
sexual assault nurse examiner => sane
small business administration, sba
securities and exchange commission, sec
southeastern pennsylvania transportation authority, septa
staff judge advocate, sja
supreme judicial court, sjc
securities litigation uniform standards act, slusa
seattle municipal code, smc
special processing unit, spu
sentence reform act, sra
social and rehabilitation services, srs
social security act, social security administration, ssa
supplemental security income, ssi
sport utility vehicle, suv
sexually violent predator, svp
tennessee bureau of investigation, tbi
trial court opinion, tco
telephone consumer protection act, tcpa
texas department of criminal justice, tdcj
truth in lending act, tila
temporary restraining order, tro
texas tort claims act, ttca
temporary total disability, ttd
texas youth commission, tyc
uniform administrative procedure act, uapa
uniform commercial code, ucc
uniform child custody jurisdiction act, uccja
uniform code of military justice, ucmj
underinsured motorist, uim
united parcel service, ups
uniform reciprocal enforcement of support act, uresa
united services automobile association, usaa
united states citizenship and immigration services, uscis
united states department of agriculture, usda
united states postal service, usps
united states trustee, ust
veterans employment opportunities act, veoa
vehicle identification number, vin
violation of probation, vop
verbatim report of proceedings, vrp
victim and witness protection act, vwpa
workers compensation appeal board, wcab
workers compensation judge, wcj
women in crisis, wic
zoning board of adjustment, zba
zoning hearing board, zhb

# Corporate synonyms from #923 
corp,corporation

# Special ones for fun
free law project, flp
court listener, cl
courthouse ethics and transparency act, ceta

My process for this was pretty manual. The document I posted earlier had all the abbreviations that the script encountered. I began by sorting them by the abbreviation and then doing a count of the rows and sorting by that.

Once the count was in it's own column, I skimmed down about 4,000 entries by eye, marking the ones that seemed good and ignoring ones that were messy or whatever. By the time I got to the ones that were only seen once or twice, I got bored and they started seeming less interesting, so I quit. Somebody could go through the rest someday and read each line making a determination if any are worth including.

With that done, I merged duplicates like...

occupational safety and health act, occupational safety and health administration, osha

...I removed punctuation, and I marked down abbreviations that are common English words (by eye).

For those, I made the synonym directional. This fixes the English language problem. It's fine, for example, if searches for "convention against torture" return documents with the word "cat" but the opposite is bad. By encoding this as...

convention against torture => cat

...we're able to make sure that the phrase returns the acronym, but not vice versa. That is, searches for cat won't return torture content. (This seems good.)

---

So that's the file we can add to the one in #2765.

abbrev-1p-1975-random-order.xlsx where I did the work (see the "Good Ones" tab).

@albertisfu, can you take it from here, please?

[mlissner]

Bill encouraged me to look for longer ones, and he was right. Here are the top acronyms longer than the last batch, but still shorter than nine characters (gotta draw the line somewhere):

comprehensive environmental response compensation and liability act, cercla
illegal immigration reform and immigrant responsibility act, iirira
financial institutions reform, recovery and enforcement act, firrea
bankruptcy abuse prevention and consumer protection act of 2005, bapcpa
uniform child custody jurisdiction and enforcement act, uccjea
religious land use and institutionalized persons act, rluipa
uniformed services employment and reemployment rights act, userra
federal employee group life insurance act, feglia
new york city employees retirement system, nycers
uniform division of income for tax purposes act, uditpa
california jury instruction criminal, caljic
florida deceptive and unfair trade practices act, fdutpa
alaska national interest lands conservation act, anilca

Here's the file: abbrev-long-1p-1975-random-order.csv

[flooie]

@mlissner - here is my argument for ten characters

Uniting and Strengthening America by Providing Appropriate Tools Required to Intercept and Obstruct Terrorism (USA PATRIOT) Act of 2001

[mlissner]

boo, hiss!

[mlissner]

@konklone shares this work from years ago: https://github.com/unitedstates/bill-nicknames

We could go through it, but I feel like I've done enough here to boost things. It seems to use more casual names in any case, which I think we probably (?) don't want to use.