search

jtleider / censusdata

Download data from Census API
MIT License
139 stars 29 forks source link

Calling 50 or more variables in a single query #7

Closed PFurst2000 closed 5 years ago

PFurst2000 commented 5 years ago

The current Census API does not allow calling 50 or more variables in a single query. Is there a solution to takes a list of variable names, and collect them into chunks with no more than 50 variables, thereby calling 50+ variables in a single query? An example of a solution in R for this issue is described at: https://rstudio-pubs-static.s3.amazonaws.com/19337_2e7f827190514c569ea136db788ce850.html

PFurst2000 commented 5 years ago

50+ calls may be performed by creating multiple calls with under 50 variables and then joining the downloaded dataframes using merge_df

Example: merge_df = pd.merge(df1, df2, left_index=True, right_index=True)

jtleider commented 5 years ago

Thanks for the suggestion, I have updated the package to accommodate downloading an unlimited number of variables at a time by chunking.

PFurst2000 commented 5 years ago

Thank you Julien. This is a huge help. Wonderful package you have developed!

On Sun, Oct 6, 2019 at 7:04 PM Julien Leider notifications@github.com wrote:

Closed #7 https://github.com/jtleider/censusdata/issues/7.

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/jtleider/censusdata/issues/7?email_source=notifications&email_token=AHPA27VFWXHL5JE5WDM3HI3QNJVIXA5CNFSM4I35WNYKYY3PNVWWK3TUL52HS4DFWZEXG43VMVCXMZLOORHG65DJMZUWGYLUNFXW5KTDN5WW2ZLOORPWSZGOUBMVLFI#event-2690209173, or mute the thread https://github.com/notifications/unsubscribe-auth/AHPA27TKEVP53Y3YLOHPFU3QNJVIXANCNFSM4I35WNYA .

jtleider commented 5 years ago

Great, glad it's helpful!

On Mon, Oct 7, 2019, 4:51 PM PFurst2000 notifications@github.com wrote:

Thank you Julien. This is a huge help. Wonderful package you have developed!

On Sun, Oct 6, 2019 at 7:04 PM Julien Leider notifications@github.com wrote:

Closed #7 https://github.com/jtleider/censusdata/issues/7.

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub < https://github.com/jtleider/censusdata/issues/7?email_source=notifications&email_token=AHPA27VFWXHL5JE5WDM3HI3QNJVIXA5CNFSM4I35WNYKYY3PNVWWK3TUL52HS4DFWZEXG43VMVCXMZLOORHG65DJMZUWGYLUNFXW5KTDN5WW2ZLOORPWSZGOUBMVLFI#event-2690209173 , or mute the thread < https://github.com/notifications/unsubscribe-auth/AHPA27TKEVP53Y3YLOHPFU3QNJVIXANCNFSM4I35WNYA

.

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/jtleider/censusdata/issues/7?email_source=notifications&email_token=ABCA7FYFVXWFJLC6HJJ5A3DQNOVOZA5CNFSM4I35WNYKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAR5Q6Y#issuecomment-539220091, or mute the thread https://github.com/notifications/unsubscribe-auth/ABCA7F77X375GQS2W7EFRCLQNOVOZANCNFSM4I35WNYA .

PFurst2000 commented 5 years ago

Hi Julien,

I am looking to revise the attached Variable Info code to handle queries from 1990 and 2000 SF3 and SF1 files. Within the code the statement of presrc = 'dec/' currently exists for ACS and 2010 SF1 queries. I need to revise the code so that presrc = null for 1990 and 2000 queries because dec/ is not part of the API web address for pre-2010 census data. So far revisions using code such as the following have all failed. Any suggestions? elif (src == 'sf1' or src == 'sf3') and year <= 2010: presrc = '' elif (src == 'sf1' or src == 'sf3') and year <= 2010: presrc = None (edited)

I understand that it may be a large undertaking to revise the code for your publicly available python package. If so, I would like to customize the package for my own personal use.

On Mon, Oct 7, 2019 at 5:51 PM Peter Furst peterthefurst@gmail.com wrote:

Thank you Julien. This is a huge help. Wonderful package you have developed!

On Sun, Oct 6, 2019 at 7:04 PM Julien Leider notifications@github.com wrote:

Closed #7 https://github.com/jtleider/censusdata/issues/7.

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/jtleider/censusdata/issues/7?email_source=notifications&email_token=AHPA27VFWXHL5JE5WDM3HI3QNJVIXA5CNFSM4I35WNYKYY3PNVWWK3TUL52HS4DFWZEXG43VMVCXMZLOORHG65DJMZUWGYLUNFXW5KTDN5WW2ZLOORPWSZGOUBMVLFI#event-2690209173, or mute the thread https://github.com/notifications/unsubscribe-auth/AHPA27TKEVP53Y3YLOHPFU3QNJVIXANCNFSM4I35WNYA .

jtleider commented 4 years ago

Hi,

Are you revising the censusvar function? presrc should be automatically set to an empty string if src is 'sf3' (see line 44), so I don't think this is your problem, at least not for SF3. If my framework for constructing the URL is not working in this case, you can revise line 45 itself.

Thank you, Julien

PFurst2000 commented 4 years ago

I finally had some extra time to work with this again and I found that I cant seem to be able to change the code from the variable_info.py file to run SF1 samples from the 1990 and 2000 datasets. As you previously mentioned, there is no issue running the code with the SF3 sample.

Original Code: if (src == 'acs1' or src == 'acs5' or src == 'acsse') and year >= 2010: presrc = 'acs/' elif src == 'sf1': presrc = 'dec/' else: presrc = ''

Proposed Code if (src == 'acs1' or src == 'acs5' or src == 'acsse') and year >= 2010: presrc = 'acs/' elif (src == 'sf1' and year >= 2010): presrc = 'dec/' elif [(src == 'sf1' or src == 'sf3') and year <= 2010]: presrc = '' else: presrc = ''

Error from proposed code: dec/ is included in proposed code when it should be omitted.

[image: image.png]

On Tue, Oct 29, 2019 at 9:03 PM Julien Leider notifications@github.com wrote:

Hi,

Are you revising the censusvar function? presrc should be automatically set to an empty string if src is 'sf3' (see line 44), so I don't think this is your problem, at least not for SF3. If my framework for constructing the URL is not working in this case, you can revise line 45 itself.

Thank you, Julien

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/jtleider/censusdata/issues/7?email_source=notifications&email_token=AHPA27V66R3KJZW3BFDTLDDQRDMMXA5CNFSM4I35WNYKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECSSPPI#issuecomment-547694525, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHPA27V473J6FEPGX5YRTB3QRDMMXANCNFSM4I35WNYA .

jtleider commented 4 years ago

It looks like the code above would set presrc = '' for 1990 and 2000 SF1 downloads. I am unclear on the error you are encountering. For me to be able to assist you, please clarify how you intend to call the function, the URL you want the function to go to, and the URL the function is going to with your modified code.

Also, please note there is a typo in your proposed code as year==2010 is included in both if conditions for src=='sf1'.