Null values from redis online store

[rajeshdhanda]

In our python feature-server (FastAPI), we have multiple projects where we fetch data from redis online store.

feast versions were upgraded from 0.18.1 to 0.26.0 BigQuery version 3.4.0 during error.

feast==0.26.0
feast[gcp]==0.26.0
feast[redis]==0.26.0

# import libraries...

fs_project1_prod = FeatureStore(repo_path=f"./{args.bucket_name}/project1/prod/")
fs_project1_stag = FeatureStore(repo_path=f"./{args.bucket_name}/project1/stag/")
fs_project2_prod = FeatureStore(repo_path=f"./{args.bucket_name}/project2/prod/")
.... so on 

feature_stores = {
    "project1_prod": fs_project1_prod,
    "project1_stag": fs_project1_stag,
    "project2_prod": fs_project2_prod,
     .... so on 
}
app = FastAPI()

@app.post("/get-online-features")
def get_online_features(body: dict=Depends(get_body)):
    try:
        feature_store = body['feature_store'] + "_" + body['env'].lower()
        store = feature_stores[feature_store]

        # read entities_to_fetch, full_feature_names from json body
        some code line ... 

        response = store._get_online_features(
            features=features,
            entity_values=entities_to_fetch,
            full_feature_names=full_feature_names,
            native_entity_values=True
        ).to_dict()

        return response
    except Exception as e:
        logging.error(str(traceback.format_exc()))
        raise HTTPException(status_code=500, detail=str(traceback.format_exc()))

we were able to fetch data from server for all , except one project showing always Null values from online store.

TTL for online store is timedelta(weeks=52) and project was created 4 months ago.

1. feast materialize-incremental with feast 0.26.0 BigQuery version 3.4.0 shows this error :

   File "/workspace/main.py", line 200, in CheckMaterialize
   fs_materialize(feast_bucket,project,env,file_name)
   File "/workspace/main.py", line 182, in fs_materialize
   fs.materialize_incremental(end_date=datetime.utcnow(),feature_views=[fv])
   File "/layers/google.python.pip/pip/lib/python3.8/site-packages/feast/usage.py", line 283, in wrapper
   return func(*args, **kwargs)
   File "/layers/google.python.pip/pip/lib/python3.8/site-packages/feast/feature_store.py", line 1325, in materialize_incremental
   provider.materialize_single_feature_view(
   File "/layers/google.python.pip/pip/lib/python3.8/site-packages/feast/infra/passthrough_provider.py", line 254, in materialize_single_feature_view
   raise e
   File "/layers/google.python.pip/pip/lib/python3.8/site-packages/feast/infra/materialization/local_engine.py", line 156, in _materialize_one
   table = offline_job.to_arrow()
   File "/layers/google.python.pip/pip/lib/python3.8/site-packages/feast/infra/offline_stores/offline_store.py", line 116, in to_arrow
   return self._to_arrow_internal()
   File "/layers/google.python.pip/pip/lib/python3.8/site-packages/feast/infra/offline_stores/bigquery.py", line 498, in _to_arrow_internal
   q = self._execute_query(query=query)
   File "/layers/google.python.pip/pip/lib/python3.8/site-packages/feast/usage.py", line 294, in wrapper
   raise exc.with_traceback(traceback)
   File "/layers/google.python.pip/pip/lib/python3.8/site-packages/feast/usage.py", line 283, in wrapper
   return func(*args, **kwargs)
   File "/layers/google.python.pip/pip/lib/python3.8/site-packages/feast/infra/offline_stores/bigquery.py", line 514, in _execute_query
   block_until_done(client=self.client, bq_job=bq_job, timeout=timeout)
   File "/layers/google.python.pip/pip/lib/python3.8/site-packages/feast/infra/offline_stores/bigquery.py", line 611, in block_until_done
   if bq_job.exception():
   File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/api_core/future/polling.py", line 282, in exception
   self._blocking_poll(timeout=timeout)
   File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/bigquery/job/query.py", line 1245, in _blocking_poll
   super(QueryJob, self)._blocking_poll(timeout=timeout, **kwargs)
   File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/api_core/future/polling.py", line 137, in _blocking_poll
   polling(self._done_or_raise)(retry=retry)
   File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/api_core/retry.py", line 349, in retry_wrapped_func
   return retry_target(
   File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/api_core/retry.py", line 191, in retry_target
   return target()
   File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/bigquery/job/query.py", line 1358, in _done_or_raise
   self._reload_query_results(retry=retry, timeout=transport_timeout)
   File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/bigquery/job/query.py", line 1329, in _reload_query_results
   api_timeout = self._done_timeout - _TIMEOUT_BUFFER_SECS
   TypeError: unsupported operand type(s) for -: 'object' and 'float'

2. Downgrading feast to 0.18.1 in new python 3.8 environment , were able to materialize and download data from redis.

   feast==0.18.1
   feast[gcp]==0.18.1
   feast[redis]==0.18.1

   but still showing Null values for one project if try to fetch data using feast==0.26.0

• feature_store.yaml for project

  project: vahan_finder
  registry: gs://bucket_name/vahan_finder/prod/registry.db
  provider: gcp
  online_store:
  type: redis 
  connection_string: IP:port, password=password
  offline_store:
  type: bigquery
  dataset: vahan_finder
  flags:
  alpha_features: true
  on_demand_transforms: true

• feature_repo.py for project

  
  from datetime import timedelta
  from feast.types import Float32, String, Int32, Int64
  from feast import BigQuerySource, Entity, Feature, FeatureView, ValueType, Field

---------------------------------- vahan ------------------------------------

vahan_fuel_type = Entity( name="vahan_fuel_type", join_keys=["qs_trans_fuel_type"], value_type=ValueType.STRING)

vahan_mmv = Entity( name="vahan_mmv", join_keys=["qs_vahan_mmv"], value_type=ValueType.STRING)

vahan_year = Entity( name="vahan_year", join_keys=[ "qs_vahan_year"], value_type=ValueType.INT32)

vahan_data_source = BigQuerySource( table="gcp_project.vahan_finder.table_name",
timestamp_field="event_timestamp", created_timestamp_column="created")

vahan_fv = FeatureView( name= "vahan_fv",entities=[vahan_year, vahan_mmv,vahan_fuel_type], ttl=timedelta(weeks=52), schema = [ Field(name="count", dtype=Int32), Field(name="car_info", dtype=String)], source=vahan_data_source)

@achals  we have downgraded our feature-server to 0.18.1 and want to upgrade it to nearest latest versions again. 

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.