BigQuery MIMIC-IV query limit exceeded error

[marinaperezs]

Hi everyone!

I am working on a project for university using the MIMIC-IV database and I have connected it to BigQuery following the steps described in the documentation (creating a project, etc). But since a few days ago, I have been getting an error in BigQuery whenever I try to run even a simple query:

I am using my university email account to work on this project so I don't know how to solve this problem, because I cannot execute queries when this error appears.

Any help is appreciated!! Thank you!

[alistairewj]

BigQuery have a free tier which allows you to make a few small queries without charge. Once you start using it more often, you are charged for usage. In order to set this up, you need to configure a project in Google Cloud Platform and associate it with a payment profile. You can read about GCP projects here: https://cloud.google.com/resource-manager/docs/creating-managing-projects

For what it's worth, querying is typically very cheap.

[marinaperezs]

Hi @alistairewj ! Thank you for your answer! I managed to run a query I need, but I need the resulting table in order to use it with Python. It is a big table (more than 1 GB) so I cannot download it to my local as a csv or excel file from BigQuery. I also tried saving it as a table in BigQuery but I don't have permissions. Do you have any ideas on how can I do this ? Because I need to access it from python in order to carry on with my project. Thank you !!

[saran-ahluwalia-patchd]

@marinaperezs One can write to table following the examples here: https://cloud.google.com/bigquery/docs/writing-results

I would not recommend saving the full data to a local computer - both because of the policies dictated in the training and, in general, in the event your computer fails you may lose access.

[marinaperezs]

@saran-ahluwalia-patchd Yes, that is what I tried, but apparently I need permissions to be able to write to table, that's why I was asking for alternative methods. Do you know how can I get the missing permissions ? Or if there's any other way of doing it ? Thanks!

[alistairewj]

You do not have permission to write to the datasets under physionet-data - including mimiciv_hosp, mimiciv_icu, etc, otherwise you could overwrite the original data. However, you can create your own dataset in BigQuery under your own project. Once you do that, you can write out the data to a table there.

[marinaperezs]

I've tried doing that but it says I need an organization to create my own project. How can I do it?

[saran-ahluwalia-patchd]

Hi @marinaperezs You need to ensure you have added permissions to read, write and delete tables in your IAM permissions: https://cloud.google.com/bigquery/docs/control-access-to-resources-iam.

Access Control Docs are here: https://cloud.google.com/bigquery/docs/access-control

Whoever administers your project needs to update your permissions if you are not able to create or write to a UDF table.

[marinaperezs]

Thank you for your help ! I'm trying to save it on a local computer in case I don't get the BigQuery permissions. Do you know how to create the sql database on a Mac ? I have to download sql and an administration tool like DBeaver but how do I create the mimic database in order to start working with it?

[alistairewj]

Without expertise that might be difficult. There are a few tutorials on the website though: https://mimic.mit.edu/docs/gettingstarted/local/

Are you a member of an organization on GCP, e.g. with your school? Otherwise I don't understand your error message. Projects can be associated with organizations, but they don't need to be.

[marinaperezs]

Hi @alistairewj ! Yes, my GCP account is with my school email, but there is not a payment profile associated, and when I try to do it I get: The user does not have the "resourcemanager.projects.createBillingAssignment" permission to link a billing account to the project.

[alistairewj]

Hrm. I'm not sure what you can do if you cannot create a payment profile. The steps you'd need to do are: (1) create a payment profile, (2) create a project, (3) create a billing ID, and (4) assign the billing ID to the project.

If you can't do any of those steps, then it might be worth it to create a personal account, and access GCP through that. You can always update your Google email in your PhysioNet profile and re-request access via BigQuery.

[marinaperezs]

I asked my school for an account with a billing profile to assign to my project and they already granted me access, so it's solved!

Thank you all for you help!!

[alistairewj]

Glad you got it sorted!