A Memento Aggregator CLI and Server in Go.
HEAD requestCommand line interface of MemGator allows retrieval of the TimeMap and the description of the closest Memento (equivalent to the TimeGate) over STDOUT in all supported formats. Logs and benchmarks (in verbose mode) and Error output are available on STDERR unless appropriate files are configured. For further details, see the full usage.
$ memgator [options] {URI-R} # TimeMap from CLI
$ memgator [options] {URI-R} {YYYY[MM[DD[hh[mm[ss]]]]]} # Description of the closest Memento from CLIWhen run as a Web Service, MemGator exposes following customizable endpoints:
$ memgator [options] server
TimeMap: http://localhost:1208/timemap/{FORMAT}/{URI-R}
TimeGate: http://localhost:1208/timegate/{URI-R} [Accept-Datetime]
Memento: http://localhost:1208/memento[/{FORMAT}|proxy]/{DATETIME}/{URI-R}
About: http://localhost:1208/about
Monitor: http://localhost:1208/monitor - (Over SSE, if enabled)
{FORMAT} => link|json|cdxj
{DATETIME} => YYYY[MM[DD[hh[mm[ss]]]]]
[Accept-Datetime] => Header in RFC1123 formatTimeMap endpoint serves an aggregated TimeMap for a given URI-R in accordance with the Memento RFC. Additionally, it makes sure that the Mementos are chronologically ordered. It also provides the TimeMap data serialized in additional experimental formats.TimeGate endpoint allows datetime negotiation via the Accept-Datetime header in accordance with the Memento RFC. A successful response redirects to the closes Memento (to the given datetime) using the Location header. The default datetime is the current time. A successful response also includes a Link header which provides links to the first, last, next, and previous Mementos.Memento endpoint allows datetime negotiation in the request URL itself for clients that cannot easily send custom request headers (as opposed to the TimeGate which requires the Accept-Datetime header). This endpoint behaves differently based on whether the format was specified in the request. It essentially splits the functionality of the TimeGate endpoint as follows:
Link header of the TimeGate response, but as the payload in the format requested by the client.Location header.proxy is used instead of a format then it acts like a proxy for the closest original unmodified Memento with added CORS headers.About endpoint reports the list of upstream archives, their status, and values of various configurations of the server.Monitor is an optional endpoint that can be enabled by the --monitor flag when the server is started. If enabled, it provides a stream of the benchmark log over SSE for realtime visualization and monitoring.NOTE: A fallback endpoint /api is added for compatibility with Time Travel APIs to allow drop-in replacement in existing tools. This endpoint is an alias to the /memento endpoint that returns the description of a Memento.
Depending on the machine and operating system download appropriate binary from the releases page. Change the mode of the file to executable chmod +x MemGator-BINARY. Run from the current location of the downloaded binary or rename it to memgator and move it into a directory that is in the PATH (such as /usr/local/bin/) to make it available as a command.
Build a Docker image locally from the source.
$ git clone https://github.com/oduwsdl/MemGator.git
$ cd MemGator
$ docker image build -t oduwsdl/memgator .Alternatively, pull a published image from one of the two Docker image registries below:
$ docker image pull docker.pkg.github.com/oduwsdl/memgator/memgator
$ docker image pull oduwsdl/memgatorRun MemGator with various options inside a Docker container.
$ docker container run -it --rm oduwsdl/memgator -h
$ docker container run -it --rm oduwsdl/memgator [options] {URI-R}
$ docker container run -it --rm oduwsdl/memgator [options] {URI-R} {YYYY[MM[DD[hh[mm[ss]]]]]}
$ docker container run -d --name=memgator-server -p 1208:1208 oduwsdl/memgator [options] server
$ curl -i http://localhost:1208/about
$ docker container rm -f memgator-server _____ _______ __
/ \ _____ _____ / _____/______/ |___________
/ Y Y \/ __ \/ \/ \ ___\__ \ _/ _ \_ _ \
/ | | \ ___/ Y Y \ \_\ \/ __ | | |_| | | \/
\__/___\__/\____\__|_|__/\_______/_____|__|\___/|__|
# MemGator ({Version})
A Memento Aggregator CLI and Server in Go
Usage:
memgator [options] {URI-R} # TimeMap from CLI
memgator [options] {URI-R} {YYYY[MM[DD[hh[mm[ss]]]]]} # Description of the closest Memento from CLI
memgator [options] server # Run as a Web Service
Options:
-A, --agent=MemGator/{Version} <{CONTACT}> User-agent string sent to archives
-a, --arcs=https://git.io/archives Local/remote JSON file path/URL for list of archives
-b, --benchmark= Benchmark file location - defaults to Logfile
-c, --contact=https://git.io/MemGator Comment/Email/URL/Handle - used in the user-agent
-D, --static= Directory path to serve static assets from
-d, --dormant=15m0s Dormant period after consecutive failures
-F, --tolerance=-1 Failure tolerance limit for each archive
-f, --format=Link Output format - Link/JSON/CDXJ
-H, --host=localhost Host name - only used in web service mode
-k, --topk=-1 Aggregate only top k archives based on probability
-l, --log= Log file location - defaults to STDERR
-m, --monitor=false Benchmark monitoring via SSE
-P, --proxy=http://{HOST}[:{PORT}]{ROOT} Proxy URL - defaults to host, port, and root
-p, --port=1208 Port number - only used in web service mode
-R, --root=/ Service root path prefix
-r, --restimeout=1m0s Response timeout for each archive
-S, --spoof=false Spoof each request with a random user-agent
-T, --hdrtimeout=30s Header timeout for each archive
-t, --contimeout=5s Connection timeout for each archive
-V, --verbose=false Show Info and Profiling messages on STDERR
-v, --version=false Show name and versionAssuming that Git and Go (version >= 1.14) are installed. Cloning, running, building, and installing the code can be done using following commands:
$ git clone https://github.com/oduwsdl/MemGator.git
$ cd MemGator
$ go run main.go
$ go build
$ go install
$ memgator --help
$ memgator http://example.com/To compile cross-platform binaries run the crossbuild.sh script:
$ ./crossbuild.shThis will generate binaries for various OSes and Architectures in /tmp/mgbins directory.
A publication related to this project appeared in the proceedings of JCDL 2016 (Read the PDF). Please cite it as below:
Sawood Alam and Michael L. Nelson. MemGator - A Portable Concurrent Memento Aggregator: Cross-Platform CLI and Server Binaries in Go. In Proceedings of the 16th ACM/IEEE-CS on Joint Conference on Digital Libraries, JCDL 2016, pp. 243-244, Newark, New Jersey, USA, June 2016.
@inproceedings{jcdl-2016:alam:memgator,
author = {Sawood Alam and
Michael L. Nelson},
title = {{MemGator - A Portable Concurrent Memento Aggregator}},
booktitle = {Proceedings of the 16th ACM/IEEE-CS Joint Conference on Digital Libraries},
series = {JCDL '16},
year = {2016},
month = {jun},
location = {Newark, New Jersey, USA},
pages = {243--244},
numpages = {2},
url = {http://dx.doi.org/10.1145/2910896.2925452},
doi = {10.1145/2910896.2925452},
isbn = {978-1-4503-4229-2},
publisher = {ACM},
address = {New York, NY, USA}
}