Closed PostAlmostAnything closed 2 years ago
@PostAlmostAnything If #20633 doesn't cover your issue, then please attach a small, runnable project or post a small, runnable code listing that reproduces what you are seeing so that we can investigate.
Thanks, I am confused by #20633. It has convinced me that there is an issue somewhere in my query probably with a Lambda expression, but it leaves me nowhere else to go.
Is there a way to make the error message say exactly where in the query it originates?
I don't know how I would go about attaching runnable code, but I could paste the query that I believe leads to the error
The query this appears to be linked to is my posts query that pulls in data from every post and every linking table. As a result the task is 155 lines long.
@PostAlmostAnything - If you can post the query and class definition of your result then I can inspect and tell you where it is getting null value rather than non-null.
The root cause of issue is when trying to read a column from database and assigning it to non-nullable property (e.g. int), if we get null then it is error because null cannot be assigned to int property. 2.1 threw error for this. 3.1 had bug and rather than throwing error it would return default value. So we decided to break because 3.1 queries while they worked gave wrong results. Wrong results from query can corrupt data which is worse than query which throws error. This error generally happens when there is mismatch in database/model (e.g. a column which is nullable mapped to non-null property), or certain query operations like left join which generates null are being assigned to non-nullable property which is bad query.
Trying to see if I under stand this correctly. You want an error to be thrown when an integer field is null? Why would you want that? I have many integer columns in my database that are null by default because users don't always need to populate them.
You should not have done this. Wrong results from a query are not worst than an error because an error breaks the site. Why not just warn the user of the possible problem rather than force people like me to do a bunch of work?
Here is the query causing the problem. As you can see it is quite big and since the error does not say exactly where in the query it occurs I feel like I am looking for a needle in a haystack.
public async Task<Posts[]> GetPosts()
{
return await (from post in _context.Posts
where post.Active == (bool?)true && post.Adminban == (bool?)false && post.Site == (int?)GlobalStatic.SITENUMBER()
select post into pts
select new Posts
{
Postid = pts.Postid,
Title = pts.Title,
Description = pts.Description,
Dateposted = pts.Dateposted,
Datemodified = pts.Datemodified,
Video = pts.Video,
Videostream = pts.Videostream,
Location = pts.Location,
Tags = pts.Tags,
Cap = pts.Cap,
Titletag = pts.Titletag,
Metatag = pts.Metatag,
Link = pts.Link,
Linkurl = pts.Linkurl,
Category = pts.Category,
Subcategory = pts.Subcategory,
Subcategory2 = pts.Subcategory2,
Worldarea = pts.Worldarea,
Region = pts.Region,
City = pts.City,
Sendemail = pts.Sendemail,
Userid = pts.Userid,
Active = pts.Active,
Adminban = pts.Adminban,
Posturl = linkgenerator.postlink($"{pts.Postid}", $"{pts.Title}"),
Comments = (from cm in pts.Comments
where cm.Active == (bool?)true && cm.Adminblock == (bool?)false
select cm into cmts
select new Comments
{
Commentid = cmts.Commentid,
Comment = cmts.Comment,
Date = cmts.Date,
Active = cmts.Active,
Adminblock = cmts.Adminblock,
Userid = cmts.Userid,
Ratings = cmts.Ratings.Select((Ratings rts) => new Ratings
{
Commentnumber = rts.Commentnumber,
Rating = rts.Rating
}).ToArray(),
Images = cmts.Images.Select((Images imgs) => new Images
{
Image = imgs.Image,
Imagename = imgs.Imagename
}).ToArray(),
Likes = cmts.Likes.Select((Likes lks) => new Likes
{
Likeid = lks.Likeid,
Commentid = lks.Commentid
}).ToArray()
}).ToArray(),
Commentcount = pts.Comments.Where((Comments cm) => cm.Active == (bool?)true && cm.Adminblock == (bool?)false).Count(),
Ratings = (from r in pts.Ratings
where r.CommentnumberNavigation.Active == (bool?)true && r.CommentnumberNavigation.Adminblock == (bool?)false
select r into rtgs
select new Ratings
{
Ratingid = rtgs.Ratingid,
Rating = rtgs.Rating,
Daterated = rtgs.Daterated,
CommentnumberNavigation = new Comments
{
Commentid = rtgs.CommentnumberNavigation.Commentid
}
}).ToArray(),
Ratingcount = pts.Ratings.Where((Ratings rc) => rc.CommentnumberNavigation.Active == (bool?)true && rc.CommentnumberNavigation.Adminblock == (bool?)false).Count(),
Ratingavg = computations.ratingavg((from rac in pts.Ratings
where rac.CommentnumberNavigation.Active == (bool?)true && rac.CommentnumberNavigation.Adminblock == (bool?)false
select rac into ra
select ra.Rating).Average()),
Images = pts.Images.Select((Images imgs) => new Images
{
Imageid = imgs.Imageid,
Imagename = imgs.Imagename,
Image = imgs.Image,
Imagetype = imgs.Imagetype,
Postid = imgs.Postid,
Comment = imgs.Comment,
Userid = imgs.Userid
}).ToArray(),
Primaryimage = computations.primaryimage((from pic in pts.Images
where pic.Imagetype == (int?)1
select pic into pi
select new Images
{
Imageid = pi.Imageid,
Image = pi.Image,
Imagename = pi.Imagename
}).DefaultIfEmpty().First()),
Likes = pts.Likes.Select((Likes lks) => new Likes
{
Likeid = lks.Likeid,
Post = lks.Post,
Commentid = lks.Commentid
}).ToArray(),
Likecount = pts.Likes.Where((Likes lc) => lc.Commentid == null).Count(),
WorldareaNavigation = new Worldarea
{
Worldarea1 = pts.WorldareaNavigation.Worldarea1,
Worldareaacronym = pts.WorldareaNavigation.Worldareaacronym,
Wurl = linkgenerator.worldarealink($"{pts.WorldareaNavigation.Worldareaacronym}")
},
RegionNavigation = new Regions
{
Regionname = pts.RegionNavigation.Regionname,
Regionacronym = pts.RegionNavigation.Regionacronym,
Rurl = linkgenerator.regionlink($"{pts.WorldareaNavigation.Worldareaacronym}", $"{pts.RegionNavigation.Regionacronym}")
},
CityNavigation = new Cities
{
City = pts.CityNavigation.City,
Cityacronym = pts.CityNavigation.Cityacronym,
Cityurl = linkgenerator.citylink($"{pts.WorldareaNavigation.Worldareaacronym}", $"{pts.RegionNavigation.Regionacronym}", $"{pts.CityNavigation.Cityacronym}")
},
CategoryNavigation = new Categories
{
Categoryname = pts.CategoryNavigation.Categoryname,
Categorylongname = pts.CategoryNavigation.Categorylongname,
Categorytitle = pts.CategoryNavigation.Categorytitle,
Categorydescription = pts.CategoryNavigation.Categorydescription,
Categorykeywords = pts.CategoryNavigation.Categorykeywords,
Categorymeta = pts.CategoryNavigation.Categorymeta,
Categoryurl = pts.CategoryNavigation.Categoryurl,
Curl = linkgenerator.categorylink($"{pts.CategoryNavigation.Categoryurl}")
},
SubcategoryNavigation = new Subcategories
{
Subcategoryname = pts.SubcategoryNavigation.Subcategoryname,
Subcategorylongname = pts.SubcategoryNavigation.Subcategorylongname,
Subcategorytitle = pts.SubcategoryNavigation.Subcategorytitle,
Subcategorydescription = pts.SubcategoryNavigation.Subcategorydescription,
Subcategorymeta = pts.SubcategoryNavigation.Subcategorymeta,
Subcategorykeywords = pts.SubcategoryNavigation.Subcategorykeywords,
Subcategoryurl = pts.SubcategoryNavigation.Subcategoryurl,
Scurl = linkgenerator.subcategorylink($"{pts.CategoryNavigation.Categoryurl}", $"{pts.SubcategoryNavigation.Subcategoryurl}")
},
Subcategory2Navigation = new Subcategory2
{
Subcategory2name = pts.Subcategory2Navigation.Subcategory2name,
Subcategory2longname = pts.Subcategory2Navigation.Subcategory2longname,
Subcategory2title = pts.Subcategory2Navigation.Subcategory2title,
Subcategory2description = pts.Subcategory2Navigation.Subcategory2description,
Subcategory2meta = pts.Subcategory2Navigation.Subcategory2meta,
Subcategory2keywords = pts.Subcategory2Navigation.Subcategory2keywords,
Subcategory2url = pts.Subcategory2Navigation.Subcategory2url,
Sc2url = linkgenerator.subcategory2link($"{pts.CategoryNavigation.Categoryurl}", $"{pts.SubcategoryNavigation.Subcategoryurl}", $"{pts.Subcategory2Navigation.Subcategory2url}")
}
}).ToArrayAsync();
You want an error to be thrown when an integer field is null? Why would you want that? I have many integer columns in my database that are null by default because users don't always need to populate them.
No that is wrong understanding. Integer columns can be null in database, EF core doesn't really care about it. But when you trying to get an integer result out in client side (integer result and not Nullable<int>
) then it is error. non-nullable integer on client side cannot take null value.
The query is materializing a lot of properties into concrete types (rather than anonymous types). Any of those types may bring in hard constraint that property is of non-nullable type. Since the access spans multiple navigation, if you are passing through optional navigation then it can become null. To identify problematic part of the query, try removing small pieces of projection one by one and execute query. It will pin-point out which part of projection is having erroneous data.
So you're saying I should just downgrade to 3.1 because what you're suggesting would take hours
Also, what is a concrete type? When I Google it all I see is concrete.
Could you maybe explain what you're talking about in plain English instead of tech gibberish?
Posts
, Comments
etc are concrete types or classes which you are initializing. They are concrete since their structure is fixed. If a property on class has type int
then it must be assigned int property for that value.
At the end of the day, what you want to do is your choice. 3.x gave wrong results. Not all wrong results causes data corruption based on how they are used afterwards. Since a null value cannot be assigned to non-null property in C# world, we have made the change and we do not intend to revert back.
In plain English:
Remove 1 line of property assignment in the final projection and re-run query, till you figure out which property assignment is exactly causing error and see for yourself that you are actually trying to assign a null value to non-nullable property. You can use ??
operator to assign some value when database returns null.
Also, how would I go about "try removing small pieces of projection one by one and execute query" without the pages breaking due parts of the query missing?
For instance, if I were to remove the line that says "Postid = pts.Postid" then every razor page relying on that query would fail because Postid would no longer exist.
I am thinking that maybe I should start by checking the models themselves and making sure that anything corresponding to a database filed that has null values using the ? such as int? instead of just int
Also, is there anything in Visual Studio that I can use to figure out exactly where in the query the error is coming from?
While if you remove the parts of projection, blazer page would fail due to data missing, but the exception message would be different and also different stacktrace which would indicate if the query failed or data display failed. Can also use debugger with break-point to see if query passes or not. Another idea would be to have a test which runs just the query (without having any page to render results which would fairly encapsulate problematic query. For non-id kind of properties, you can also remove query fragment and just assign some random value based on type so at that point the particular property assignment doesn't remain dependent on data.
Model approach would be good for simple query. In complex cases it may not always work. If an entity has non-null property and all the data is also non-null in database (which means it is good) but the entity is being used in query through an optional navigation and if that navigation doesn't exist, (i.e. no matching record), then a non-null column in database can still return null value. The latter is the scenario where generally this kind of error shows up.
Looks like the only non-nullable integer types in my Posts model other than the primary key which has no null values are Commentcount, Ratingcount, and Likecount but they also don't have default values. Those three are not mapped to any database tables and their values are computed by the task. I don't think these are the problem because when the counts are zero their value is zero not null.
In linked models for linked tables (comments, ratings, images, category, subcategory, worldarea, region, city) the only non-nullable integer types are primary and foreign keys with no null values in the database.
Your second to last sentence said "If an entity has non-null property and all the data is also non-null in database (which means it is good) but the entity is being used in query through an optional navigation and if that navigation doesn't exist, (i.e. no matching record), then a non-null column in database can still return null value." That has me thinking that maybe the problem is related to linked tables that don't always have values for each post like images, ratings, comments, and likes.
Could the following part of the query result in a null value if there are zero items:
Likes = pts.Likes.Select((Likes lks) => new Likes { Likeid = lks.Likeid, Post = lks.Post, Commentid = lks.Commentid }).ToArray(),
Likeid is a non-nullable type because it is the primary key of the Likes table and cannot be null. My understanding is that if there are no matching likes for a post then an empty array is returned, I could be wrong. If a post has no likes then does it result in a null value for Likeid?
If the collection navigation is enumerated and has no records, then it will return empty array. It wouldn't cause above error. Btw, the error could happen for any non-nullable type not just int.
Primaryimage = computations.primaryimage((from pic in pts.Images
where pic.Imagetype == (int?)1
select pic into pi
select new Images
{
Imageid = pi.Imageid,
Image = pi.Image,
Imagename = pi.Imagename
}).DefaultIfEmpty().First())
Is there matching record for this in database or is the result default value? We have some edge case issues around DefaultIfEmpty in some cases.
It returns the first matching record from the database or a default value if no matching record exists. So, if a post has no matching images it will return values of "No image" for Imagename and "https://example.com/images/no-image.png" for Image. This is the function for computations.primaryimage:
public static Images primaryimage(Images primaryimage) { if (primaryimage.Imageid == 0) { primaryimage.Imagename = "No Image"; primaryimage.Image = GlobalStatic.SITEIMAGES() + GlobalStatic.SITENOIMAGE(); } return primaryimage; }
I'm not sure how an Imageid gets a value of 0. I checked the database in SSMS and found no default value or binding for Imageid. Nor is there a default value assigned in the model or in the razor pages (ex: if Imageid is null then Imageid = 0). Is it possible that the empty image id is being set to null and not zero?
Is Images
struct? If it is a class then ideally the result should be null (default value for a reference type) if there are not matching images. EF Core has a bug in special scenario in 5.0 where it wouldn't return the null (true default value) but rather than instance of Images
with all property with their null values.
Looking at above function you shared, looks like it doesn't check for null either, so looks like there is always matching value in which case DefaultIfEmpty is not really needed with current data (so trying running query with it and see if it works in 3.1 or not). or you are running into bug around DefaultIfEmpty and were getting wrong result (the custom function actually works based on wrong result) and now it tries to give correct result but fails because types don't match. In this case, you can coalesce all properties of Images to default value so Imageid = pi.Imageid ?? 0
and likewise.
P.S. would have played around with query myself and found what is causing issue and what should be the correct query but looking at size of the query, I assumed that providing a runnable repro code would be difficult.
The images model looks like this:
public partial class Images
{
[Key]
public int Imageid { get; set; }
public string Image { get; set; }
public int? Imagetype { get; set; }
public int? Postid { get; set; }
public int? Comment { get; set; }
public string Userid { get; set; }
[MaxLength(50)]
[RegularExpression("^(?!.*<[^>]+>).*", ErrorMessage = "No HTML tags allowed!")]
[DataType(DataType.Text)]
public string Imagename { get; set; }
public virtual Comments CommentNavigation { get; set; }
public virtual Imagetypes ImagetypeNavigation { get; set; }
public virtual Posts Post { get; set; }
}
But Primaryimage is defined in the Posts model because it only applies to posts. A post can have zero or more images.
[NotMapped] public virtual Images Primaryimage { get; set; }
If I recall correctly, back when I wrote the primaryimage function I tried checking for a null id at first, but later learned that for whatever reason the default value was 0.
I tried rewriting it by creating a default image object and then using that instead of the Primaryimage function, but I got a giant error message that says "DefaultIfEmpty(__Defaultimg_0)' could not be translated. Either rewrite the query in a form that can be translated, or switch to client evaluation explicitly by inserting a call to 'AsEnumerable', 'AsAsyncEnumerable', 'ToList', or 'ToListAsync'. See https://go.microsoft.com/fwlink/?linkid=2101038 for more information."
The changes I made included adding this above the first line that begins with "return await":
Images Defaultimg = new Images { Imageid = 0, Imagename = "No Image", Image = GlobalStatic.SITEIMAGES() + GlobalStatic.SITENOIMAGE() };
Then I altered the part that sets the primary image to say:
Primaryimage = (from pic in pts.Images
where pic.Imagetype == (int?)1
select pic into pi
select new Images
{
Imageid = pi.Imageid,
Image = pi.Image,
Imagename = pi.Imagename
}).DefaultIfEmpty(Defaultimg).First(),
Obviously I don't want to translate a single image object into a list.
I just tried rewriting the primaryimage function to check for null Imageid but I get a green squiggly saying that the value will always be false because type int can never be null. If that is the case then how is it possible for an int to be null in the first place, I'm guessing that might have something to do with why the default value of an int in an Image object would be zero.
Also, if the problem is with a non-nullable type not having a value then why does it say "Nullable object must have a value" doesn't that mean that the object without a value is not nullable?
Wrapping the primaryimage function in if (primaryimage != null){} with an else that creates a new image changes nothing
My debugger output says Microsoft.EntityFrameworkCore.Query: Error: An exception occurred while iterating over the results of a query for context type 'PostAlmostAnything.Data.ApplicationDbContext'. System.InvalidOperationException: Nullable object must have a value. at lambda_method159(Closure , QueryContext , DbDataReader , ResultContext , SingleQueryResultCoordinator )
How do I figure out where lambda_method159 is?
Here is the SQL from the output for the images. I noticed it says SELECT NULL AS [empty], is that the problem?
OUTER APPLY (
SELECT TOP(1) [t].[imageid], [t].[image], [t].[imagename], 1 AS [c]
FROM (
SELECT NULL AS [empty]
) AS [empty]
LEFT JOIN (
SELECT [i].[imageid], [i].[image], [i].[imagename]
FROM [images] AS [i]
WHERE ([p].[postid] = [i].[postid]) AND ([i].[imagetype] = 1)
) AS [t] ON 1 = 1
) AS [t0]
Alright, If I make Imageid a nullable type in the Images model and change the primaryimage function to check for Imageid == null instead of Imageid == 0 and then assign primaryimage.Imageid = 0 to the object then it works.
This solution still leaves a lingering question, why did the last version of .Net Core default to Imageid == 0 instead of null?
Also, could making a primary key a nullable type have complications for other parts of my app like auto increment when inserting new images?
The "fix" in my last comment broke something else. Now I cannot remove some of the images. The following method no longer works for removing images from the database:
From my /edit/Index.cshtml.cs file:
[BindProperty]
public int?[] DeleteImages
{
get;
set;
}
public Images Images
{
get;
set;
}
... Then this is from my OnPostAsync task:
if (DeleteImages != null && DeleteImages.First() != 0)
{
foreach (var img in DeleteImages)
{
Images imgdelete = new Images();
imgdelete = _context.Images.Where((Images p) => p.Imageid == img).First();
_context.Images.Remove(imgdelete);
await _context.SaveChangesAsync();
}
}
From my edit/Index.cshtml file:
@if (Model.Posts.Images.Count > 0)
{
<div class="form-group">
@foreach (var img in Model.Posts.Images.Where((Images hi) => hi.Imagetype == 1))
{
<div id="imgWrapper_@img.Imageid">
<button type="button" class="close mb-2" aria-label="Close" onclick="deleteimg(@img.Imageid)"><small aria-hidden="true">×REMOVE</small></button>
<img id="imgResult_@img.Imageid" name="imgResult-@img.Imageid" src="@img.Image" alt="@img.Imagename" class="img-fluid w-100 rounded shadow-sm mx-auto d-block lazyload" />
<div class="row mb-3 mt-3">
<label id="imgLabel_@img.Imageid" for="imgName_@img.Imageid" class="col-sm-2">Name Above Image:</label>
<div class="col-sm-7">
<input id="imgName_@img.Imageid" name="imgName-@img.Imageid" onchange="reValidateForm();" class="form-control" type="text" data-val="true" data-val-maxlength="Max 50 Chars." data-val-maxlength-max="50" data-val-regex="No HTML tags allowed!" data-val-regex-pattern="^(?!.*<[^>]+>).*" maxlength="50" value="@img.Imagename" />
</div>
<span id="imgError_@img.Imageid" data-valmsg-for="imgName-@img.Imageid" data-valmsg-replace="true" class="text-danger col-sm-3 col-form-label"></span>
</div>
</div>
}
</div>
}
From the javascript file that populates the delete images array:
var deleteimgs = $.makeArray();
function deleteimg(imgid) {
$('#imgWrapper_' + imgid + '').remove();
deleteimgs.push(imgid);
$('#DeleteImages').val(deleteimgs);
reValidateForm();
}
The above is of course not the entire file, but they are what I think are the parts responsible for image removal. I built it a long time ago, so I can't be sure exactly. I could have forgotten some parts.
Since the update it won't remove the rows from the database when the form is submitted. The idea was to render all images assigned to a specific post on a form. Then if someone clicked a button to remove any of them then the image id would be extracted from the element and added to an array of integers. Then when the form is submitted any rows in the images table with matching id numbers is removed. It worked just fine before this update. I'm not sure if this is directly tied to having to change the primary key of the Images model to an int?
On the client side everything goes fine. I just checked the header and found that the entire array of image ids to remove were posted to the form properly.
Now I remember how I made this work. The javascript populates the value field of an input element on the page. That element should be bound to the DeleteImages array defined in the page model, but for some reason .Net is no longer threating the value of the element as an array anymore. I figured this out by doing some tinkering and that is the only explanation why it accepts single values, but throws a validation error whenever there are more than one value. It also explains why DefaultImages.First() != 0 did not work because there is no array to get First() from.
The element on the form looks like this:
<input asp-for="DeleteImages" type="hidden" value="0" />
Now the question is, how do I make it treat the value of the element as an array again?
@PostAlmostAnything I've gone over the above, and at this point it's difficult to make out exactly what your code looks like, what your original problem was, or what you need help on.
I suggest trying trying to concentrate on the EF Core-related problems you're running into, and to exclude any ASP.NET, Javascript, or other unrelated code; your last question seems to be unrelated to EF Core (though I can't be sure), and these extra layers only make it harder to focus on your exact EF issue and gain a clear picture of the situation.
The best way forward here is to put together a minimal code sample - ideally a simple console program - with the minimal EF model and query which trigger your problem. From experience, while trying to create such a minimal code sample, users frequently realize what was wrong in their original program; and if not, that code sample would provide us with the exact info we need in order to help you.
Trying to see if I under stand this correctly. You want an error to be thrown when an integer field is null? Why would you want that? I have many integer columns in my database that are null by default because users don't always need to populate them.
You should not have done this. Wrong results from a query are not worst than an error because an error breaks the site. Why not just warn the user of the possible problem rather than force people like me to do a bunch of work?
I don't think that what he's saying. I've had the problem described.
Table Employee
Id int, not null
Age int, Allow Nulls
Class Employee int Id int Age <-- here's the error in the class's attribute (it should be int? to match the database table definition)
When EF tries to assign the value of null to this, it has a choice. Use the default value of 0, or throw an error because the database has a null but the class does not allow it.
In 2.1 they through an error. In 3.0 the team used the default value of 0 (which is way bad--because the class definition does not allow null). So they reverted back to 2.1 and now throw an error when there is a type mismatch.
EF Team Triage: Closing this issue as the requested additional details have not been provided and we have been unable to reproduce it.
BTW this is a canned response and may have info or details that do not directly apply to this particular issue. While we'd like to spend the time to uniquely address every incoming issue, we get a lot traffic on the EF projects and that is not practical. To ensure we maximize the time we have to work on fixing bugs, implementing new features, etc. we use canned responses for common triage decisions.
I just got done upgrading to .Net 5.0 and now my app returns the following error whenever I run it:
System.InvalidOperationException: 'Nullable object must have a value.'
I have not changed the queries used on the home page at all since last running the app. What changes might cause this? I do have a number of nullable types used in the query for which the values passed from the home page are null, but isn't the whole point of using nullable types so that you can pass null values?
I also get a pending database migration notice. I've never used a code first approach so I always make changes to my database using SSMS before updating the entity models by hand. Is there any way to make this notice go away other than running the migration?