garethbrickman
commented
1 month ago Could you try using print() or context.log.info() to see and confirm the EnvVar value for start_date?
Closed EvanZ closed 1 month ago
garethbrickman
commented
1 month ago Could you try using print() or context.log.info() to see and confirm the EnvVar value for start_date?
EvanZ
commented
1 month ago Hi @garethbrickman could you suggest where to log to? I added a logging command to the asset that is using the partition but I don't see any logs being created. I'm not even sure where to put the print statement either to be honest.
EvanZ
commented
1 month ago While I wait for reply, I should note that when I use os.environ instead of EnvVar to define the partition, it works without issue:
import os
from datetime import datetime
from dagster import DailyPartitionsDefinition, EnvVar
daily_partition = DailyPartitionsDefinition(
start_date=os.environ['START_DATE'],
end_date=os.environ['END_DATE']
)So it must be something about the timing of loading .env that I am not understanding.
garethbrickman
commented
1 month ago You could create a test asset in your code just to print/log the values and materialize it. Here's a self-contained example:
from dagster import asset, OpExecutionContext, Definitions, EnvVar
import os
@asset
def test_asset(context: OpExecutionContext):
context.log.info(f"Using EnvVar: {EnvVar('START_DATE').get_value()}")
context.log.info(f"Using os.getenv: {os.getenv('START_DATE')}")
defs = Definitions(
assets=[test_asset]
)
- When os.getenv is used, the variable's value is retrieved when Dagster loads the code location and will be visible in the UI.
- When EnvVar is used, the variable's value is retrieved at runtime and won't be visible in the UI.
EnvVar defers resolution of the environment variable value until run time, and should only be used as input to Dagster config or resources. To access the environment variable value, call get_value on the EnvVar, or use os.getenv directly.
EvanZ
commented
1 month ago Thanks @garethbrickman. I have to admit that part of the docs confused me a bit when I read it before. When I launch the Dagster UI via dagster dev, for example, is that "run time"? Or is "run time" when an asset is materialized? For example, if I run dagster dev materialize some assets, and then decide to change environment variables, is it the case that EnvVar will not see that change? And that if I want changes in env variables to be propagated to asset materializations, I should use os.getenv?
garethbrickman
commented
1 month ago I think "run time" is when an asset is materialized/job is run, it's referred to also as "launch time" as in launching a job with a configuration.
In terms of changing the contents of the .env file, any time the file is modified the workspace must be re-loaded to make the Dagster webserver/UI aware of the changes.
Dagster version
dagster, version 1.7.14
What's the issue?
I am working with partitions by date. In my partition file I have the following code:
I am getting the environment variables from my
.env:It seems that when I run
dagster devthe env variables are being loaded because I see the following being logged to stdout:2024-07-31 15:17:06 -0700 - dagster - INFO - Loaded environment variables from .env file: DUCKDB_DATABASE,START_DATE,END_DATEHowever, I get the following error traceback:
I can't figure out what is causing this behavior.
What did you expect to happen?
In a previous version of my code I had defined the dates in a constants file, so essentially I was hardcoding them. I want to transition to using environment variables.
How to reproduce?
No response
Deployment type
Local
Deployment details
No response
Additional information
No response
Message from the maintainers
Impacted by this issue? Give it a 👍! We factor engagement into prioritization.