search

orma / openroads-vn-analytics

BSD 2-Clause "Simplified" License
0 stars 1 forks source link

RLP upload failures #545

Closed geohacker closed 5 years ago

geohacker commented 5 years ago

@PCDuc and @kai2ser reported that there are RLP upload jobs that are failing or freezing with no errors. This ticket outlines a way to identify the problems, and fix them.

@batpad and I look at the logs during upload attempts and came across a few things:

  1. Some malformed coordinates
  2. Some invalid GeoJSONs
  3. In total there have been 45 RLP uploads. These are not unique vpromm uploads. Just number of jobs created.

The following vpromms have failed as we can see:

232CL00008 232HT00001 214TT00057 033HG00002 034HG00004 034HG00006 034HG00007 034HG00027 034HG00012

Next steps:

cc @batpad @olafveerman

geohacker commented 5 years ago

Kai requested for job timestamps, and which ones are successful. Here's a list of all the jobs, vpromm ids, and timestamp. rlp-jobs.xlsx

geohacker commented 5 years ago

Some more questions from Kai with regards to the log:

What do the "null" attempts likely represent?

null in the vpromm id field means the RLP upload didn't specify a vpromm id.

The summary shows that there we only 9 identified uploads?

Based on the logs from January 15, this is what we can see.

The successful one was done two times?

Likely, yes.

Getting the RLP successfully ingested?

Yes, I think we're missing something in terms of the actual RLP files being attempted versus what we used for writing this API and testing. So we can proceed debugging once you share the files.

Seeing who well the RPL attributes overlays on the WoN? (This depends on alignment of RPL to WoN)

Right, this process has been outlined in Duc's last email. We've also captured the entire process of expected conflation here https://github.com/orma/openroads-vn-analytics/issues/505#issuecomment-435478469 At the moment, we'll focus on fixing the RLP upload itself.

batpad commented 5 years ago

It was great to get the sample files, thanks @PCDuc - and I think we've been able to identify the root causes of the errors and create tickets.

Here's a summary of issues we've been able to identify so far:

Inconsistencies between CSV files

The actual errors were mostly small inconsistencies between CSV files, column ordering and names, etc. These are mostly easy to fix once we have examples of what these inconsistencies are. Specifically, any data inconsistency that caused invalid geometries to be created during the CSV parsing process was causing errors that were not caught cleanly. This is issue is tracked here https://github.com/orma/openroads-vn-analytics/issues/547

Error handling

The larger problem that surfaced was that we need to improve some of our error handling - these issues are fundamentally related to the fact that the RLP Upload process now does a lot more computationally expensive work to deal with conflation, and we have switched to using a background worker process to handle this. We need to fix the way we handle certain kinds of errors in this background process, and also make consistent the way errors get displayed to users. Ticket: https://github.com/orma/openroads-vn-analytics/issues/548

Graceful job termination

A third, related problem is that when we have uncaught errors, it causes a bit of a cascade of failures since it causes the background worker process to not process new messages cleanly (hence things stuck in a "Processing" state). Ticket here: https://github.com/orma/openroads-vn-analytics/issues/549

So, the actual fixes for data inconsistencies are fairly trivial, but we would want to also include the other fixes mentioned above so that in the future we have better error handling in case more data inconsistencies arise, and also ensure that errors in one upload do not in any way affect other uploads, which currently seems to be an issue.

It's super helpful to have more files to test with, and we expect to be able to resolve these issues by next week.

tailm1 commented 5 years ago

@geohacker Please help to reopen this bug. we will meet this defect.

) for reference.

tailm1 commented 5 years ago

Add @PCDuc to follow

geohacker commented 5 years ago

@tailm1 thank you for the comment. Unfortunately, that ZIP file you shared doesn't seem to be valid. We're seeing the following when we try to open. It creates a cpgz file, and then another zip file but none of them have any CSV data.

image

This is probably the reason why ORMA isn't allowing to upload so could you double check the file and get back to us?

Lastly, can you outline the differences between RLP Android and RLP iOS? This is the first time we're learning this issue and our assumption was that the data specification will be the same regardless of the mobile platform. cc @olafveerman @kpgeo

tailm1 commented 5 years ago

@geohacker Could you tell me what is expected files can we use for uploading. Please share 1 sample with only mandatory fields. I want to make sure this problems come from RLP because the attached files is just exported by RLP.

KPGeo commented 5 years ago

@tailm1 To our knowledge, there should not be a difference between the files generated by RLP on Android and iOS. Is there some local compression tool that is in use there?

@geohacker do the files still work if the .cpgz extension is removed?

PCDuc commented 5 years ago

@geohacker I'm using Window 10 and Winrar software to open the zip and it works well. Is there anything to do with Window vs MacOS or Winrar vs other unzip softwares? I can see first difference between RLP Android csv and RLP iOS csv is column names in RLP iOS csv have uppercase letter while RLP Android csv don't. Not sure if your algorithm is case-sensitive? Thanks

olafveerman commented 5 years ago

For whatever it's worth, I am not able to open the ZIP file on Linux. Neither with 'Archive Manager', nor with unzip in the command line:

Archive:  Ha.Giang.20190312.zip
  End-of-central-directory signature not found.  Either this file is not
  a zipfile, or it constitutes one disk of a multi-part archive.  In the
  latter case the central directory and zipfile comment will be found on
  the last disk(s) of this archive.
unzip:  cannot find zipfile directory in one of Ha.Giang.20190312.zip or
        Ha.Giang.20190312.zip.zip, and cannot find Ha.Giang.20190312.zip.ZIP, period.
geohacker commented 5 years ago

Is there anything to do with Window vs MacOS or Winrar vs other unzip softwares?

@PCDuc there shouldn't be, but this particular example has a corrupt ZIP file. This can be due to many reasons. I'd recommend simplifying the CSV -> ZIP process. Put all RLP CSVs into a single folder and ZIP the directory once. We've seen before in the sample shared by you that there was rar files inside zip files etc and that makes it really difficult to manage these files.

ORMA doesn't support RAR, so you shouldn't be using .rar format. The expected format is a ZIP file with a set of CSV files.

Regarding column structure, we will need to see a sample first to confirm this is the case. Currently we use column names exactly like how they are in the sample files shared previously. Example - https://github.com/orma/openroads-vn-api/blob/master/services/rlp-geometries.js#L64

PCDuc commented 5 years ago

Hi @geohacker, thank you for looking at this. Are you able to open the data which sent by Hang? I also prepared a file which in .zip format and has no sub-zip folder. Ha Giang (20190312).zip We're aware of requirement for ORMA input data therefore always upload data with zip format only and no zip within zip. Thanks