Exploratory data analysis

[Juan-Mateos]

This might include some questions and comments for @jaklinger and @russwinch

[Juan-Mateos]

Issues found during data exploration:

• [ ] Variables with information about the field / sector for a patent such as ipc, nace2, tech_field are almost completely missing (~100% of values are np.nan)

[jaklinger]

Which dataframe?

[bishax]

Observations pulled from notebooks in https://github.com/nestauk/patent_analysis/tree/90818d26a083af91fa5a37b355deb5ab546d6ee1/notebooks :

{'person_ctry_code': 'GB',
'earliest_filing_year': 2010,
'database': 'patstat_2019_05_13'}

person_appln table

• han_name appears to be missing universities E.g. psn_name is 'UNIVERSITY OF CAMBRIDGE' but han_name is 'CAMBRIDGE ENTERPRISE LTD'.

• ~54% (297K entries) of person_address missing

  • Some of these can be inferred by looking at non-null person_address entries for the same psn_name but a different application
  • This results in 83K psn_name's with missing addresses

• Often person_address is just a city/town

• Many names have multiple addresses

  • This often appears to be due to the same address being written many ways therefore we can just extract the postcodes (get postcodes for ~21% of entries)

• the link between person_id and person_name is unclear

  • match at patent appln id level and decide what to do w/ missing addresses?
  • ~allocate missing addresses randomly based on address distribution for persons with the same name (or id?)~

appln Table

• In almost all cases earliest_filing_year <= earliest_publn_year
• Applications in multiple jurisdictions => focusing on families helps avoid double counting patents (https://en.m.wikipedia.org/wiki/Priority_right)
• All patents in a family receive the same number of citations

appln_abstract table

• almost all patents are in english
• some are incredibly long
• mentions for terms such as "financ" and "machine learning" are low

appln_ipc table

• ipc_version has different dates - what does it mean for IPC data to be available at different points
• 42240 unique IPC class symbols
• ipc_class_level almost exclusively "A" (full IPC codes)
• The IPC symbols are not available in a standardised format. We convert them by replacing spaces with 0s, and /s with ''s
• Most codes are ten digits. Occasionally they are longer, where we shorten them to ten digits.