Nicholas-Westby
commented
3 years ago I don't see anything in the code that would cause a failure on CSV export after a few thousand records. Is there some code you can share that creates the data so I can reproduce it locally?
Looking at the stack trace, perhaps one of the field values is null and maybe the CSV writer doesn't like to write null values? There was at least at some point exactly that bug (though I think it was fixed for the version of CsvHelper we're using): https://github.com/JoshClose/CsvHelper/issues/275
Here's the body of the Formulate function that writes the values in case you want to review it for issues:
/// <summary>
/// Generates a CSV of the form submissions for the specified form.
/// </summary>
/// <param name="form">
/// The form to generate the CSV for.
/// </param>
/// <returns>
/// The CSV.
/// </returns>
private byte[] GenerateCsvOfFormSubmissions(Form form)
{
// Variables.
var formId = form.Id;
var db = ApplicationContext.DatabaseContext.Database;
var filteredFields = form.Fields.Where(x => !x.IsTransitory);
var fieldIds = filteredFields.Select(x => x.Id).ToArray();
var fieldsById = filteredFields.ToDictionary(x => x.Id, x => x);
var extractedValues = new List<SubmissionExport>();
// Query database for form submissions.
var query = new Sql().Select("*").From("FormulateSubmission")
.Where("FormId = @0", formId).OrderByDescending("CreationDate");
var entries = db.Fetch<FormulateSubmission>(query).ToList();
// Extract field values from the database entries.
foreach (var entry in entries)
{
var extractedRow = new Dictionary<Guid, string>();
var exportRow = new SubmissionExport()
{
Submission = entry,
Values = extractedRow
};
extractedValues.Add(exportRow);
var values = GetValuesForFields(entry.DataValues);
var filenames = GetValuesForFiles(entry.FileValues);
foreach (var fieldId in fieldIds)
{
var value = default(string);
if (values.TryGetValue(fieldId, out value))
{
}
else if (filenames.TryGetValue(fieldId, out value))
{
}
else
{
value = null;
}
extractedRow[fieldId] = value;
}
}
// Store to a CSV.
var config = new CsvHelper.Configuration.CsvConfiguration()
{
HasHeaderRecord = true,
QuoteAllFields = true
};
using (var memStream = new MemoryStream())
{
// Open stream/writer.
using (var textWriter = new StreamWriter(memStream))
using (var writer = new CsvWriter(textWriter, config))
{
// Write headers.
writer.WriteField("Creation Date");
writer.WriteField("URL");
writer.WriteField("Page ID");
foreach (var fieldId in fieldIds)
{
var field = fieldsById[fieldId];
writer.WriteField(field.Name);
}
writer.NextRecord();
// Write values.
foreach (var row in extractedValues)
{
writer.WriteField(row.Submission.CreationDate);
writer.WriteField(row.Submission.Url);
writer.WriteField(row.Submission.PageId);
foreach (var fieldId in fieldIds)
{
var value = default(string);
if (!row.Values.TryGetValue(fieldId, out value))
{
value = null;
}
writer.WriteField(value);
}
writer.NextRecord();
}
}
// Return CSV as a byte array.
return memStream.ToArray();
}
}
TFAstudio
Hey folks,
We're load testing a solution right now, and after a few thousand records are added via a data handler, the CSV Download interface fails in the back-office. Yes it works after just a few records are added. Yes the records in the database seem fine and we can page through them in the back-office. The front end form does not fail. Just the CSV download back-office process after the testing is done and we try and download the data.
We do not know exactly what the record tipping point is. But it seems volume related. Umbraco v7.15.7, Formulate v2.6
Any ideas?
The CSV download loads an error in the same window: