vacancy / SceneGraphParser

A python toolkit for parsing captions (in natural language) into scene graphs (as symbolic representations).
MIT License
546 stars 55 forks source link

Doesn't work with attributes #19

Open ifmaq1 opened 2 years ago

ifmaq1 commented 2 years ago

Hi, I tried to generate the graph for sentences like

a white and red toothbrush
a black computer monitor

however the parser doesn't work with these sentences. I get it, this parser works with subject relation and object. Can you please guide me, how do I manage these lines to generate graph for such sentences.

Thank you

vacancy commented 2 years ago

The second sentence should work:

In [2]: sng_parser.parse("a black computer monitor")
Out[2]:
{'entities': [{'span': 'a black computer monitor',
   'lemma_span': 'a black computer monitor',
   'head': 'computer monitor',
   'lemma_head': 'computer monitor',
   'span_bounds': (0, 4),
   'modifiers': [{'dep': 'det', 'span': 'a', 'lemma_span': 'a'}],
   'type': 'unknown'}],
 'relations': []}
vacancy commented 2 years ago

Due to 14ed400

The first sentence should also work now.

In [1]: import sng_parser

In [2]: sng_parser.parse('a white and red toothbrush')
Out[2]:
{'entities': [{'span': 'a white and red toothbrush',
   'lemma_span': 'a white and red toothbrush',
   'head': 'toothbrush',
   'lemma_head': 'toothbrush',
   'span_bounds': (0, 5),
   'modifiers': [{'dep': 'det', 'span': 'a', 'lemma_span': 'a'},
    {'dep': 'amod', 'span': 'white', 'lemma_span': 'white'},
    {'dep': 'amod', 'span': 'red', 'lemma_span': 'red'}],
   'type': 'unknown'}],
 'relations': []}
ifmaq1 commented 2 years ago

Hi, Thanks for your response. For relations table, it was easy to create graph by picking the words from relation's table. Like object relation and subject

how to extract the information for graphs through entities table when we can't have relations' table. I want to create this graph.

image

For the a black computer monitor sentence. I want one node to be Blackand other node to be computer monitor. Just like white and table in above image

Also, It couldn't detect a man is in the water & the man is laying down in relation's table. Although these sentences include relation in it.

vacancy commented 2 years ago

Hi,

There are two unresolved cases in the previous model:

For "the man is laying down". Currently, the model does not handle any intransitive verbs. I don't think they should be treated as "relations".

ifmaq1 commented 2 years ago

here are some more sentences it couldn't detect

umbrella is blue and white. microwave oven in the kitchen. stainless steel oven red fire hydrant in snow bear with black eyes green wall behind the tennis court the bears head the water is blue. umbrella is open. the water is calm. sky is clear and blue. two teddy bears on a shelf. the train is in the station.

hbai98 commented 1 year ago

One case from my tests. "A rainforest bird mating ritual dance." {'entities': [{'head': 'rainforest bird', 'lemma_head': 'rainforest bird', 'lemma_span': 'a rainforest bird', 'modifiers': [{'dep': 'det', 'lemma_span': 'a', 'span': 'a'}], 'span': 'a rainforest bird', 'span_bounds': (0, 3), 'type': 'unknown'}, {'head': 'dance', 'lemma_head': 'dance', 'lemma_span': 'ritual dance', 'modifiers': [{'dep': 'amod', 'lemma_span': 'ritual', 'span': 'ritual'}], 'span': 'ritual dance', 'span_bounds': (4, 6), 'type': 'unknown'}], 'relations': [{'lemma_relation': 'mate', 'object': 1, 'relation': 'mating', 'subject': 0}]}

It interprets 'dance' as an entity, which should not be.