[249] Crunchbase v3 --> v4 API change and speedups - Githubissues

nestauk / old_nesta_daps

[archived]

MIT License

18 stars 5 forks source link

[249] Crunchbase v3 --> v4 API change and speedups #296

Closed jaklinger closed 4 years ago

jaklinger commented 4 years ago

Closes #199

Run

luigi --module crunchbase_root_task RootTask

from nesta/nesta/core/routines/datasets/crunchbase.

Changes:

[x] Variable naming changes between v3 and v4
[x] Debug new warning (described in #249)
[x] Iterate on naming of non-org collection task
[x] Change & migration of SQL ORMs
[x] propagate new fields (see below) to ES

Summary of field changes:

Organisations

old	new	Change type	New type
aliases	alias1	new field	VARCHAR(200)
	alias2	new field	VARCHAR(200)
	alias3	new field	VARCHAR(200)
funding_rounds	num_funding_rounds	rename
funding_total_us	total_funding_usd	rename
	legal_name	new field	VARCHAR(200)
	total_funding	new field	BIGINT
	total_funding_currency_code	new field	VARCHAR(3)
	num_exits	new field	INTEGER
	postal_code	new field	VARCHAR(20)
	rank	new field	INTEGER

Degrees

old	new	Change type	New type
	cb_url	new field	VARCHAR(200)
degree_id	id	rename
	institution_name	new field	VARCHAR(200)
	name	new field	VARCHAR(100)
	permalink	new field	VARCHAR(100)
	person_name	new field	VARCHAR(100)
	rank	new field	BIGINT
	type	new field	VARCHAR(50)

Acquistions

old	new	Change type	New type
state_code	acquiree_state_code	rename
	cb_url	new field	VARCHAR(200)
acquisition_id	id	rename
	name	new field	VARCHAR(100)
	permalink	new field	VARCHAR(100)
	rank	new field	BIGINT
	type	new field	VARCHAR(50)

Funding Rounds

old	new	Change type
funding_round_id	id	rename
	name	new field
investor_ids	lead_investor_ids	rename
investor_names		removed
post_money_currency_code	post_money_valuation_currency_code	rename
	type	new field
	rank	new field
company_id	org_id	rename
company_name	org_name	rename
	permalink	new field

Funds

old	new	Change type
fund_id	id	rename
fund_name	name	rename
	raised_amount_usd	new field
	cb_url	new field
	entity_name	new field
	entity_type	new field
	permalink	new field
	rank	new field
	type	new field

Investment partners

old	new	Change type
	funding_round_name	new field
	investor_name	new field
	name	new field
	id	new field
	partner_name	new field
	permalink	new field
	rank	new field
	type	new field
	updated_at	new field
	cb_url	new field
	created_at	new field

Investments

old	new	Change type
	funding_round_name	new field
	id	new field
	investor_name	new field
	investor_type	new field
	name	new field
	permalink	new field
	rank	new field
	type	new field
	updated_at	new field
	cb_url	new field
	created_at	new field

Investors

old	new	Change type
country	country_code	rename
investor_name	name	rename
investor_type	investor_types	rename
	total_funding	new field
	total_funding_currency_code	new field
	created_at	new field
	linkedin_url	new field
	permalink	new field
	rank	new field
	type	new field

IPOs

old	new	Change type
company_id	org_id	rename
company_state_code	state_code	rename
country	country_code	rename
ipo_id	id	rename
price	share_price	rename
price_currency_code	share_price_currency_code	rename
price_usd	share_price_usd	rename
	type	rename
	valuation_price	new field
	valuation_price_currency_code	new field
	valuation_price_usd	new field
	money_raised	new field
	money_raised_currency_code	new field
	org_cb_url	new field
	org_name	new field
	permalink	new field
	rank	new field

People

old	new	Change type
job_id	id	rename
	name	new field
	org_name	new field
	permalink	new field
	person_name	new field
	rank	new field
	type	new field
	updated_at	new field
	cb_url	new field
	created_at	new field

Jobs

old	new	Change type
primary_affiliation_organization	featured_job_organization_name	rename
primary_affiliation_title	featured_job_title	rename
primary_organization_id	featured_job_organization_id	rename
	name	new field
	permalink	new field
	rank	new field
	region	new field
	type	new field