Open wentaoxu415 opened 1 year ago
The README has a troubleshooting section.:
If you are using the default Dagster storage backed by SQLite, you may encounter an error as:
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) unable to open database file
To get your pipeline successfully running, you can Shift+click "Materialize all" on the asset graph page and add the following configuration. This will turn off the default multiprocessing execution.
execution:
config:
in_process: null
And I did that configuraiton and I'm seeing the same error as above. Edit: Seems to fail after about 200 tokens for me.
Hi! Thanks for opening this issue, and trying out this example!
The readme explains a workaround that may still not fit larger repository (i.e. longer running step).
The long-term solution is to switch to use a more performant storage, such as the built-in Postgres storage. To do that, you'll need to:
dagster-postgres
DAGSTER_HOME/dagster.yaml
present. if not, you can set the environment variable DAGSTER_HOME
to be where you want to store Dagster instance information. (when it's not set, Dagster tools will use a temp directory for storage that will be cleaned up when the process exists). More details in: https://docs.dagster.io/deployment/dagster-instance#default-local-behaviorstorage
in your DAGSTER_HOME/dagster.yaml
to switch to use postgres instead of the default sqlite. here's the doc: https://docs.dagster.io/deployment/dagster-instance#dagster-storage
Hi Dagster team,
Congratulations on launching this blog post! I really liked the creativity behind the analysis and went ahead to try the simple model analysis against various other sample repos. However, I ran into some issues so I wanted to file this issue and see if there are steps that we can take to overcome it.
Issue Whenever I select repos with over 200 stars, the Dagster run fails at the
stargazers_with_user_info
step because ofsqlalchemy.exc.OperationalError: (sqlite3.OperationalError) unable to open database file
error.I suspect that when the Github API request takes some time, the database connection might be getting dropped which prevents Dagster from writing debug events and trigger this issue.
Error Message
Reproduction Steps Specify any repo with more than a few hundred number of stars (> 200-300 stars) in the ops config.