search

isaacmg / fb_scraper

FBLYZE is a Facebook scraping system and analysis system.
Apache License 2.0
64 stars 21 forks source link

Need new Avro Schema for following format #11

Open isaacmg opened 7 years ago

isaacmg commented 7 years ago

We need an Avro schema that can deal with something like the following: {'from': {'name': 'Elizabeth Austen', 'id': '10212551456802570'}, 'link': 'https://www.facebook.com/events/259740027809063/permalink/276873696095696/', 'created_time': '2017-04-21T16:35:29+0000', 'type': 'link', 'name': 'Daniel', 'id': '1043983518950523_1649940175021518', 'shares': {'count': 1}, 'reactions': {'data': [], 'summary': {'total_count': 2, 'viewer_reaction': 'NONE'}}, 'comments': {'data': [], 'summary': {'order': 'chronological', 'total_count': 0, 'can_comment': False}}, 'group_id': '1043983518950523', 'reacts': {'data': [{'id': '10155178339184493', 'type': 'LIKE'}, {'id': '612725696831', 'type': 'LIKE'}], 'paging': {'cursors': {'before': 'TlRVNE1EVTVORGt5T2pFME9USTRNREl3TlRZANk1qVTBNRGsyTVRZAeE13PT0ZD', 'after': 'TVRJd01EQXdNVEF5T2pFME9USTNPVE13TlRnNk1qVTBNRGsyTVRZAeE13PT0ZD'}}}}

isaacmg commented 7 years ago

Temporarily reverted back to old JSON message in order to keep Kafka running. As soon as we have a proper schema will use this. Any help is appreciated.