Update to latest FamPlex

[MihaiSurdeanu]

Hi @bgyori and @johnbachman,

This is about updating bioresources to the latest FamPlex: https://github.com/sorgerlab/famplex I think I will need some help from you so I don't screw things up.

First, let me explain what we use from this data in bioresources:

1. We have a list of known complex names in BEcomplexes.tsv (names can be changed). The format is one name per line. Each line has: name \t grounding-id. For example:

   (9-1-1) complex 9_1_1
   9-1-1   9_1_1
   9-1-1 complex   9_1_1
   9_1_1   9_1_1
   A2t     Annexin_II_heterotetramer
   ACC     ACC
   ...

2. We have a list of protein families in BEfamilies.tsv in the same format:

   14-3-3 proteins p14_3_3
   4EBP    EIF4EBP
   a-adrenoreceptor        ADRA
   a-cathenin      CTNNA
   ABL     ABL_family
   ...

3. We have an "override" file of grounding ids, which is preferred over all KBs. This file includes some entries from FamPlex (formerly bioentities). The file is called NER-Grounding-Override.tsv, and its format is: name \t grounding-id \t originating-KB \t type. In this file we have several entries from you, which were added to override protein names. For example:

   E2F     E2F             be      Family
   ERK     ERK             be      Family
   Fos     FOS_family              be      Family
   GST     GST             be      Family
   INS     INS             be      Family
   p38     p38             be      Family
   ...

[MihaiSurdeanu]

What is the best way to re-generate these files from FamPlex? It seems that grounding_map.csv is the closest. But I do not see an obvious way to identify the types of these entities. We need to pull out the complexes and families in separate files, as mentioned above.

Also, we probably should update the override file as well. Do you know which of these complex/family names overlap with Uniprot? And which should be prioritized over Uniprot? Thanks!

[johnbachman]

Hi Mihai, it would be pretty straightforward to generate these files from the grounding map--basically we would just iterate over the entries in the grounding map and apportion the entries into the BEcomplexes, BEfamilies, and overrides files, depending on whether the string in the grounding map is grounded to a family, complex or other (protein, chemical, metabolite, biological process, etc.). We can add the generation of these files to our FamPlex export process.

[MihaiSurdeanu]

That would be awesome. Thanks!

[bgyori]

One thing to consider is that the distinction between families and complexes is somewhat blurred in FamPlex. There are entries in FamPlex that are both families and complexes (e.g., NFkappaB is a family of complexes). If there isn't an important reason to separate families and complexes, we could just use a single flat list. Otherwise, it would be useful to better understand where the family/complex distinction plays a role in REACH to be able to implement the export logic appropriately.

[MihaiSurdeanu]

The distinction is not important to Reach. Event rules operate over parent types of both complexes and families, so they are not affected by the confusion. So, if it doesn't matter to you, it doesn't matter to us. Maybe we can create a new type for both, e.g., ComplexOrFamily?

[bgyori]

Here is the first iteration of the export:

• famplex.tsv (to replace BEcomplexes.tsv and BEfamilies.tsv): https://github.com/bgyori/famplex/blob/reach_export/export/famplex.tsv
• famplex_groundings.tsv (to be added to / replace Override.tsv): https://github.com/bgyori/famplex/blob/reach_export/export/famplex_groundings.tsv

Some potential issues to consider:

• Not all entries are guaranteed to have the correct type. The export assigns BioProcess as the default type to all GO and MESH entries. But some entries may actually be diseases, cellular components, etc.
• There are some ambiguities in the grounding map, e.g.,

  ER»·ESR»fplx»···FamilyOrComplex                                                 
  ER»·GO:0005783»·go»·Simple_chemical                                             
  ER»·P03372»·uniprot»Gene_or_gene_product                                        

  what should we do with these for the purpose of this export?

• I am not totally sure what the current overrides are (there are several files in the repo with overrides) but my guess is, it will have redundancies, and possibly conflicts with this export. How should these be resolved?

[MihaiSurdeanu]

Thanks @bgyori !

A few comments:

• The format of famplex.tsv should be "text-name-of-entity \t grounding-id" (see example above in this thread). It seems your format is "grounding-id \t type". Can you please adjust it to the former?
• The "override" grounding file is slightly more costly to apply at runtime. Our convention was that we use it sparingly, only when really needed. That is, to solve ambiguities for entities that overlap with other entities in other KBs with lower priority such as Uniprot. Would it be possible to generate a smaller one, keeping just the names that overlap with others, and moving everything else to dedicated files per type? That is, we should have a separate file for each type in FamPlex, e.g., FamPlex-FamilyOrComplex.tsv, FamPlex-Simple_chemical.tsv, FamPlex-Gene_or_gene_product.tsv, etc.
• In fact, the override file was designed specifically to solve ambiguities such as your "ER" example. In such cases, it is your/our call to decide which type takes priority. This is clearly less than ideal, because we would choose the same type for instances of "ER" regardless of the context in which they appear. But it is what we have today...

[MihaiSurdeanu]

Let me know if anything in the above is not clear!

[bgyori]

Got it, will fix these issues and some others I found.

[bgyori]

A few more things that come to mind:

• Is there a simple way to get a list of all strings covered by different resources all at once, to be able to determine where an explicit override from FamPlex is needed? Or should I process each file separately to get the list of strings?
• I noticed that there is a NER stoplist in bioresources (https://github.com/clulab/bioresources/blob/master/src/main/resources/org/clulab/reach/kb/ner_stoplist.txt), we have a similar list in INDRA to handle irrelevant strings being picked up as named entities, largely from REACH: https://github.com/sorgerlab/indra/blob/master/indra/resources/grounding/ignore.csv, should we port some of these over into your ner_stoplist.txt?

[MihaiSurdeanu]

• Unfortunately, we don't have a single list of strings...
• Yes, please update ner_stoplist! Thanks!

[bgyori]

This is done, though restricted to importing only families/complexes from FamPlex. Separately, some of the non-family/complex grounding overrides could be imported into the overrides list here.