search

GLEIF-IT / lei-rdf

RDF LEI Project
MIT License
8 stars 4 forks source link

Prepping for deploying to d.w env #178

Closed laconc closed 4 years ago

laconc commented 4 years ago
laconc commented 4 years ago

I didn't realize that there were updated versions of the scripts in that branch. Let's merge the Issue-146 branch first, and then we can merge the dockerize branches on top; the only changes worth keeping from them are those needed for ECS and decryption. Redirecting to null is gross here but those commands are generating an excessive amount of log entries. I stopped the task early at one point and there were already 35k entries. Those logs go to our Datadog account and I'm concerned about the cost. I see that you're now redirecting the error output to L1-xsl.log. That's great but that file won't be available for analysis after the ECS instance dies. Not sure how we should proceed there.

rivettp commented 4 years ago

The L1 processing output is the large one, and I have an issue #162 to hopefully address that. I think it's best to keep that going to a separate file even if we lose it and have all the other processing going to stdout. When the L1 issue is addressed we should be able to use stdout for that too. OR have a flag to suppress these warning messages.