Closed AndriiLesiuk closed 1 year ago
@AndriiLesiuk could you share your data model classes and IEdmModel
configurations as well?
What version of EF Core are you using and what provider library are you using to connect EFCore to Snowflake (I'm not familiar with CData).
My guess is that either the Snowflake does not support starts/endswidth functions, or the specific DB or EF core driver you're using doesn't know how to translate these functions to the corresponding Snowflake functions.
So just checked the docs, Snowflake DOES support those functions:
What EF Core provider are you using?
@habbes Thanks for the reply!
My model class:
using System.ComponentModel.DataAnnotations;
using System.ComponentModel.DataAnnotations.Schema;
namespace ODataPrePOC.Domain.Models
{
[Table("NORM_CHEMICALCAPACITYBYCOMPANY_CAPACITYTOPRODUCE_API_DATA")]
public class NormChemicalCapacityByCompanyCapacityToProduceApiData
{
[Key]
[Column("SOURCE_ID")]
public string? Source_Id { get; set; }
[Column("ORIGIN_ID")]
public string? Origin_Id { get; set; }
[Column("CITY")]
public string? City { get; set; }
[Column("REGION_NAME")]
public string? Region_Name { get; set; }
[Column("REMARKS")]
public string? Remarks { get; set; }
[Column("START_DATE")]
public string? Start_Date { get; set; }
[Column("END_DATE")]
public string? End_Date { get; set; }
[Column("LAST_UPDATE_DATE")]
public string? Last_Update_Date { get; set; }
[Column("DATAGROUP")]
public string? Data_Group { get; set; }
[Column("CONCEPT")]
public string? Concept { get; set; }
[Column("PRODUCT")]
public string? Product { get; set; }
[Column("PROCESS")]
public string? Process { get; set; }
[Column("PRODUCER")]
public string? Producer { get; set; }
[Column("STATE")]
public string? State { get; set; }
[Column("UNIT")]
public string? Unit { get; set; }
[Column("FREQUENCY")]
public string? Frequency { get; set; }
[Column("OBSERVATIONS")]
public string? Observations { get; set; }
[Column("COUNTRY_TERRITORY")]
public string? Country_Territory { get; set; }
}
}
as EF Core provider (connector between EF and Snowflake) I use a paid solution from CData: https://www.cdata.com/kb/tech/snowflake-ado-codefirst.rst , because I couldn't find another working solution for EF and Snowflake.
My context class:
namespace ODataPrePOC.Infrastructure.DatabaseContext
{
using Microsoft.EntityFrameworkCore;
using ODataPrePOC.Domain.Models;
using ODataPrePOC.Domain.Models.RAWModels;
public class DEVELOPSchemaDbContext : DbContext
{
public DEVELOPSchemaDbContext(DbContextOptions<DEVELOPSchemaDbContext> options)
: base(options)
{ }
public DbSet<NormChemicalCapacityByCompanyCapacityToProduceApiData>? NormChemicalCapacityByCompanyCapacityToProduceApiData { get; set; }
}
}
and I didn't do any additional settings related to IEdmModel.
Cheers
And I use Microsoft.EntityFreamworkCore(6.0.15) as well.
Maybe you could check with cdata whether they have support for contains
, startswith
and endswith
functions. Worth noting that the exception thrown when using contains
is different from the other methods.
I haven't tried this, but maybe it could provide worthwhile: what about trying to create custom function mappings and translations as suggested by the exception message.
Following this guide: https://learn.microsoft.com/en-us/ef/core/querying/user-defined-function-mapping you could create a custom startswith
function and map it to the appropriate SQL or create a corresponding db procedure for it, then check if that helps.
I have no choice but to implement my own logic for building expressions:
internal static class FilterHelper
{
public static (string fieldName, string fieldValue) GetFilterParts(string filter)
{
if (string.IsNullOrWhiteSpace(filter))
{
return (null, null);
}
var filterParts = filter.Split(new[] { "(", ",", ")", " " }, StringSplitOptions.RemoveEmptyEntries);
if (filterParts.Length != 3 || filter.IndexOf("'") == -1)
{
return (null, null);
}
var fieldName = filterParts[1].Trim().ToLower();
var fieldValue = filterParts[2].Trim('\'');
fieldValue = fieldValue.TrimStart('\'');
fieldValue = ProcessFieldValue(filter, fieldValue);
return (fieldName, fieldValue);
}
public static Expression<Func<T, bool>> GetFilterPredicate<T>(string filter)
{
if (string.IsNullOrWhiteSpace(filter))
{
return null;
}
var filterParts = filter.Split(new[] { " and ", " AND ", "&&" }, StringSplitOptions.RemoveEmptyEntries);
var parameterExpression = Expression.Parameter(typeof(T), "record");
Expression<Func<T, bool>> predicate = null;
foreach (var part in filterParts)
{
var (fieldName, fieldValue) = GetFilterParts(part);
if (!string.IsNullOrWhiteSpace(fieldName) && !string.IsNullOrWhiteSpace(fieldValue))
{
var memberExpression = Expression.Property(parameterExpression, fieldName);
var constantExpression = Expression.Constant(fieldValue, typeof(string));
var methodInfo = typeof(DbFunctionsExtensions).GetMethod(nameof(DbFunctionsExtensions.Like),
new[] { typeof(DbFunctions), typeof(string), typeof(string) });
var callExpression = Expression.Call(methodInfo, Expression.Constant(EF.Functions), memberExpression, constantExpression);
var expression = Expression.Lambda<Func<T, bool>>(callExpression, parameterExpression);
predicate = predicate == null ? expression : Expression.Lambda<Func<T, bool>>(Expression.AndAlso(predicate.Body, expression.Body), parameterExpression);
}
}
return predicate;
}
public static string ProcessFieldValue(string filter, string fieldValue)
{
if (filter.IndexOf("contains", StringComparison.OrdinalIgnoreCase) >= 0)
{
return $"%{fieldValue}%";
}
if (filter.IndexOf("startswith", StringComparison.OrdinalIgnoreCase) >= 0)
{
return $"{fieldValue}%";
}
if (filter.IndexOf("endswith", StringComparison.OrdinalIgnoreCase) >= 0)
{
return $"%{fieldValue}";
}
return fieldValue;
}
}
and my service class:
public async Task<IQueryable<NormChemicalCapacityByCompanyCapacityToProduceApiData>> Get(string? filter)
{
var predicate = FilterHelper.GetFilterPredicate<NormChemicalCapacityByCompanyCapacityToProduceApiData>(filter);
if(predicate == null) return _database.NormChemicalCapacityByCompanyCapacityToProduceApiData.AsQueryable();
var filteredData = await _database.NormChemicalCapacityByCompanyCapacityToProduceApiData.Where(predicate).ToListAsync();
return filteredData.AsQueryable();
}
I'm trying to send a request like this: https://localhost:44394/odata/NormChemicalCapacityByCompanyCapacityToProduceApiData?$filter=contains(REGION_NAME, 't')
But I get the following error:
The same goes for this "endswith" and "startswith" functionality: https://localhost:44394/odata/NormChemicalCapacityByCompanyCapacityToProduceApiData?$filter=endswith(REGION_NAME, 't') https://localhost:44394/odata/NormChemicalCapacityByCompanyCapacityToProduceApiData?$filter=startswith(REGION_NAME, 't')
My Program class:
My controller class:
and my Service Get method:
The problem is solved by a change(ToList()) in the controller:
But it does not suit me, because then all the data is loaded, and filtering will take place on the client side, not on the database side. I also found this article: https://github.com/DevExpress/DevExtreme.AspNet.Data/issues/428
I work with Snowflake via CData. I will be grateful for help or advice.