garyfeng
commented
2 years ago Still don't know why this not working on this particular notebook, but the work around is to find the IP of your host machine and manually set it in the notebook. See the updated Spark Integration Demo notebook for instructions.
import os
from dotenv import load_dotenv
load_dotenv() # add this line
MARQUEZ_HOST = os.getenv('MARQUEZ_HOST')
MARQUEZ_PORT = os.getenv('MARQUEZ_PORT')
OL_URL = 'https://{}:{}'.format(MARQUEZ_HOST,MARQUEZ_PORT)
# for whatever reason using the env var is not working THIS notebook, for Spark integration only
# this works on another project; and this OL_URL works using `maraquez_client`. Go figure.
# For now, manually find the IP of your host machine (ifconfig or ipconfig for Windows) and find your IP
# You should be able to see Jupyter notebook by replacing your "localhost" or "127.0.0.1" with that IP.
OL_URL = "http://192.168.86.39:5000"
print('OpenLineage URL = {}'.format(OL_URL))
The Spark Integration notebook worked before, but is now complaining about not being able to emit OL messages. Likely a docker network issue. Need to investigate and add more details.