webrecorder / warcit

Convert Directories, Files and ZIP Files to Web Archives (WARC)
https://pypi.python.org/pypi/warcit
Apache License 2.0
81 stars 13 forks source link

Appends .csv to the log file string #30

Open Shrinks99 opened 1 year ago

Shrinks99 commented 1 year ago

This will ensure that log files are read in the proper format and that I stop opening it in my text editor! ;)

Changes

tw4l commented 12 months ago

Other thing to consider: Should we just expect the full filename with the extension to the --log argument? And if there's an issue with another tool that's using this library and isn't doing that, we could handle it upstream.

Something to think about. Sometimes a little less "magic" ends up being more flexible for a wide variety of use cases (and a little less logic to handle).

despens commented 12 months ago

My sense is that auto-appending file extensions to provided names only makes sense if the same base name is used for several output files. For instance, if a crawler would ask for a "collection name" and then produce a filename.warc.gz, filename.cdxj, and filename.log or similar.