search

codelibs / fess

Fess is very powerful and easily deployable Enterprise Search Server.
https://fess.codelibs.org
Apache License 2.0
994 stars 166 forks source link

Indexing of Pdf files From Uri - No Search Results #1345

Closed zaryk closed 5 years ago

zaryk commented 6 years ago

My goal is to extract the text from pdfs at certain urls location to be indexed. Right now I am trying to just index them without extracting the text. The urls look like this:

URLs: https://test.site.gov/Web/Orders

Include for crawling: https://test.site.gov/Web/Orders/*

Include for indexing: https://test.site.gov/Attachments/Order/*

Pdf Url Example: https://test.site.gov/Attachments/Order/476e7ac6-3dd3-42c8-8c71-c1f0b17fca43

Fess seems to be finding the urls, and looks to be "Storing child urls". Later in the log, it starts to say things such as:

No indexes are added. Searching "*" or "filetype:pdf" doesn't bring back results.

Logs:

2017-11-16 17:27:46,103 [Crawler-20171116172730-1-1] DEBUG Storing child urls: [RequestData [method=GET, url=https://test.site.gov/Web], RequestData [method=GET, url=https://test.site.gov/Web/Dockets], RequestData [method=GET, url=https://test.site.gov/Web/Ndi], RequestData [method=GET, url=https://test.site.gov/Web/Agendas], RequestData [method=GET, url=https://test.site.gov/Efile/Home], RequestData [method=GET, url=https://test.site.gov/MyDms/Login], RequestData [method=GET, url=https://test.site.gov/Web/Matters], RequestData [method=GET, url=https://test.site.gov/Web/OrderIndex], RequestData [method=GET, url=https://test.site.gov/Web/Calendar], RequestData [method=GET, url=https://test.site.gov/Web/Email], RequestData [method=GET, url=https://test.site.gov], RequestData [method=GET, url=http://www.psc.sc.gov], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/476e7ac6-3dd3-42c8-8c71-c1f0b17fca43], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/82ff0ad8-f096-4607-a6a5-765191303bb8], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/573b98c2-03b6-444c-be33-a4cffe4642f6], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/5c00fb37-c5a8-48b6-8e53-8192aa3b13d7], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/24c37632-3ce2-4783-824f-398334956607], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/2c8dd5db-2b7f-42fc-8683-393f66815779], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/805450eb-ec56-4666-b180-5e607c4dd9f1], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/6a262d2f-afe1-4abc-a540-16873333a1b9], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/05241100-d46c-4b87-b10c-d4fa5bad4bb7], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/31fe156f-74df-4a13-b958-40c1d06b75e4], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/525b9b59-a9d4-42f8-95e0-968cc0f5f540], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/02121982-5660-498f-8018-9b368787ca88], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/17978b9e-057d-4888-a8cf-c608b7de294a], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/381b403c-2a22-456a-bcc3-aafc2bed1a42], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/954ccda8-2dcd-435b-9881-71afc754ed09], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/698b2955-3249-4b17-aa89-1432286389ce], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/3a301c5a-049d-4d64-8034-8b9940423ff2], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/58bbb89d-d5ce-43d2-8c14-31034bb5873b], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/5bd1ab95-f23f-4159-8f62-20de98bf5773], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/1da3a95d-a8e0-4e79-b1de-596ce0b38a5f], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/a032ac33-5a87-42ba-97db-3aa6fe981314], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/7b09d45c-c531-481d-9414-628908f35e7c], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/58a59040-abb7-4aae-8d71-c8d44e69e401], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/cdf729bf-e36a-4941-9853-1ba74d5305ee], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/99bafaa9-5f4c-47b2-9a1a-3b3b601138ec], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/3c966eaf-fd78-4472-bf15-134bcee08258], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/b0504022-ce48-4914-bd7a-b0201ff5465e], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/81d09b8a-2f29-4bca-b19d-97acaa2591db], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/5ea952d2-0e54-4344-9352-b9178b8425c6], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/b09f810c-8dba-4e24-8e0f-ce6934d3f527], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/dbef8fe9-0068-4b51-9561-38e6b28f31fc], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/44ad504a-bbda-4c5b-9fd1-098225ac326f], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/e35164a6-4736-4dd9-9419-a0faa3ff06c4], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/ecb83317-7349-4635-83a7-346dde72435d], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/25c7703f-4847-4e1e-88b4-7e1e0bbcd265], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/eb7e8948-fd27-48c4-b540-055e9124b9f0], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/717b8512-3fa9-42d8-a952-9e2844d9382c], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/f7e28d19-252a-4f0b-8a66-3474e70067bc], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/1bc706a9-6d10-4d63-be56-4b0b76b07ce6], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/7b4dc55a-6860-47ed-99b7-836bcdd00176], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/2d414529-d053-4a44-a8ca-effd7da42100], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/fbafa811-bcd8-4f82-996a-511df968fe50], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/d0a48a3f-3c30-4080-9a27-2773abd59b40], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/742ab22a-f5ab-4b16-829a-6f0bd031406a], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/fcf34d56-f638-4ea3-b8df-5b11d0f8fe1a], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/7b686dd4-e9ab-4c45-8e5c-1af0e595f08e], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/4e71e44b-c2e6-4c4c-8040-d2ba7ffc5b95], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/08d682bf-8040-4763-b7b8-d2e28f649d3e], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/9a781a87-9677-4830-98ad-cf680e4c708c], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/ed41bcd3-bc57-4e92-af2d-329a7b32f61a], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/be920d6c-1025-4d10-8f34-f6d4c47d91ba], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/9cb03089-24ca-491a-aebc-720b83f4a83e], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/f1791e3d-b1bb-49ed-ad71-8bdf22799ccd], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/0c101a5f-90fe-434e-aac0-58a16e3a89cb], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/ae49e19e-d957-429c-9fdc-16aeba584a9b], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/4448b998-11a9-4110-b576-c58cf2103d9e], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/a557fb06-3f0e-431e-a6c4-cb2f91a27676], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/db9f041f-a54e-4069-a99d-7938e8738cb1], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/d7e9f105-42d5-48d3-9c52-bb5509371d8b], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/26f18f4f-bd2e-4fdc-bd12-d4f0b9970110], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/3ef6a6f8-6082-48f6-bb15-646c50459a32], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/c7fe4c38-2c9f-4e96-a04d-717b2aa63d4d], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/62218232-49d5-4720-84a7-ff0ce08d84ac], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/2e6b1d0d-3d7b-4542-af9c-a41881b683b6], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/0fd63d52-a712-4d1a-839e-ad93f6e0d68b], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/d22506b7-7380-438d-9f73-c39154c4222c], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/cf88b4d8-af71-4ddb-8529-d54bc28656c8], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/710c8266-6189-4cc7-99d3-82ecfb59fd7a], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/e2ad4ea0-9e5a-4610-943c-51f5f7b8dae0], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/0d0dffd9-d75b-4ca9-9427-331cd6c47203], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/21ad802c-0fc7-44c7-a56f-2c006604ef3b], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/53a2342b-f506-46c9-88fd-24a4a0e374c9], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/57ac69d7-aecb-4045-a960-cdab21090710], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/be669acf-2d38-46d9-98d8-9e2fbc86026f], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/e60bc00f-54d1-42a0-9fcc-dabf90b256a4], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/fee91ed5-a169-4c47-8a6c-2b736bfdd1d8], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/5fd9b610-40d6-490a-9ee7-48237f46a50f], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/efc732c5-b517-4ad1-bcc4-8bda34e00b2b], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/36878eb6-f18e-4f38-a4e0-a0d0bee6dc50], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/08da0543-0e68-46f2-8764-e6c166cb21c5], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/1bdc5b54-4b78-427e-98ad-ee96d9d06596], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/27969a81-c947-49b1-9d53-02bd80792f56], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/31b1c920-9a9e-4c0a-bbb9-f5ac424f5e56], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/2f53508b-81df-4670-a920-d45347958724], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/a7675cf8-0bb6-4b82-9add-d1e43bf2cd16], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/27f8ebd9-ec49-4049-a019-9814917f7270], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/d6ecb1ff-5e82-46a3-881f-84efb3e62397], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/0d66f781-7240-4d47-b5ee-8988b2e76dde], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/dee4e06d-25d2-4935-9101-8f725e6c72c7], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/7bafc54e-0d25-41aa-964c-e89976b82eef], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/1698b81a-7554-4bbb-ae1b-f15aca589421], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/e3b90ccd-bb0d-40f7-ae44-e47c78cf3d29], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/aca4bc5c-95b8-415a-a7d5-5692a9bf3077], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/19aec33e-3d95-40fd-888f-cf7e5a737782], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/926f7569-c74f-44c9-af80-90141638fc16], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/78ea4542-eb6f-4a16-8466-b130d0249ca8], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/ddba1245-92b9-4ab4-a459-57056b64d145], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/51186aa4-8a50-457c-81d1-95fe24761143], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/840d324f-6e9b-4524-b3da-4bc1a4bceecb], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/4c44a58c-c301-4661-ba41-b8e36115127f], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/2fd6ce51-7cc8-4465-9020-9a3415a63a3b], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/410da977-1d43-4386-a1fa-ae0fccccaeb8], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/922894b0-c60a-4891-a041-d2a48516ab22], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/2c066ec9-e1c1-416c-9a20-b5570cf2812e], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/99700888-f7d9-4c35-bbf4-7beac567a0f0], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/cc757ef6-79ce-4dc0-9d66-21063025cd66], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/0e3af53d-77b9-40ad-b13d-31470a6e1688], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/5fbb6c30-26e4-4835-baf7-e614eb27bd0a], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/ab780850-d990-4f6a-84b7-fa5376984696], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/eb7cc211-6ab0-41f9-ba38-6d53c1de132a], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/61769184-9ac2-436a-8010-29067edc9d95], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/545ab8fa-be40-471d-9e7c-fe17333041f7], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/c2c7b5d6-a6f8-4a0c-a7b0-1f38503bb31e], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/2d65e664-224f-45af-a4a9-f3c662d9672d], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/b29111dd-e035-483d-a895-515ecc7f7623], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/1e188959-90a4-422c-a51e-b2b87c8825d1], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/e62740e7-3f65-4821-9854-be8d4aeff56a], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/987dece0-ec19-45f6-90f5-b05618a7b7f4], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/3e4cc2dd-1ee4-48ae-8151-45e816c587da], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/77c6d8da-d9a9-418b-84b9-133cbb991797], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/dc0e4f81-9b78-48a8-ac73-af522f2c072d], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/b7e84a35-bde2-427b-9d0d-f3215656c477], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/595382ce-0081-41f5-be35-7788de410e6d], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/4e15e364-0503-474d-b096-b261a25ac9b1], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/1fbdd79f-4b4d-45c7-8e0c-bef5a6db9a32], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/f1306a4c-90ca-4683-aef9-ef24e3058da1], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/4375d634-65ac-48c3-b9ba-9fb7076cb564], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/80b62d95-5d30-4e2d-a19a-8cd56d63b98b], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/048b6ed3-eaf3-4b55-9069-627485dd2965], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/874d89da-596d-4597-8e4d-a40577c76005], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/426275c4-f8a4-4367-a9c9-f74ccfecfed7], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/e19fe604-e293-4281-8cf6-7655d0c33651], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/674b0e5d-1ced-4cd9-a5e0-149dc3f58ad1], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/fb4605a9-8c70-4afa-bb7c-84bed5a9f7e8], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/1be4a7e5-f0df-4939-a05a-04e6740f51de], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/3212f384-acc0-45a2-88b5-8c9a3279dcb1], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/9487c7e0-0864-4658-ae27-01a1a988da45], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/401738dc-a337-4693-9719-2af41c60d35d], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/860246b1-ced6-41e4-bb8f-ec453a98679d], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/88bd5df0-c040-4ead-b0ee-6efaf74415c8], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/8770451f-3a08-4774-bc05-157d764080cd], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/13f57bc6-09d4-42f1-a060-92fa1e3f6554], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/84358209-ff8a-4196-ac6a-ee872d9a47d9], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/0c9ce90e-6c2f-4ebe-b14c-1baa5ac31d51], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/6a83f5e9-6ff9-4a3e-9de6-554d0aa4df7c], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/934d3ec1-7023-41ba-9216-6012066fc641], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/025a7ed1-e406-4c8e-8194-f1957b9d7e3d], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/66b7d100-1708-406b-a170-25e785aea2c2], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/fe9ee9db-a3bd-4dc0-b684-8f254bef5f80], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/8270b538-8363-4c01-8735-f8dbf23fc590], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/72903433-e660-439d-97ac-fa5d66ede19f], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/142b7c0b-e39e-47ca-a3c0-62d7f70aa017], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/7733a32c-302a-4e08-be77-e47d45b057df], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/e6beda07-6f28-4305-996f-4d3d64993297], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/24cafa91-aee5-4c47-aa54-f8ae0aa92892], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/a6d681fc-9242-45f1-a1a1-be9a313abe97], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/f6e399d7-9d71-44ae-8e6e-b531313f818b], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/de1b2ded-5078-4e6a-9028-65d1fd327041], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/4e6e9c4c-b533-445e-a364-4f184912a047], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/71568509-ac78-44e5-bd22-6968115feff8], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/a85de289-f0e8-4d36-bcc1-66f76b03855e], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/405a2ee7-e8a5-48d8-8d8c-dc1edc64ce2a], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/6656eab3-d27f-4f31-b033-34825ff1e10b], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/60718b4d-c38f-4dc7-ad21-9e341a70c380], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/7293e4d6-79ad-40f7-ae46-2eb6e4d19599], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/df9ede02-7210-4882-ad48-38318fb4939f], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/1552b67e-2dd0-4258-a1c0-d9c59544cd59], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/5b71c1c4-0af1-4d81-b359-af8ac973f7fe], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/3a735967-9e76-4df8-aa40-01358bef0e85], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/efa32767-9442-46f2-8bd4-b79958dd9063], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/eac50227-8d63-4466-8ebb-096e5f7ca4da], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/3eaec112-54fd-49cb-81bb-4f64dc59c9d9], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/6bf91e62-cd13-4cb4-9757-efd96ec21963], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/f59f1a5f-b6d2-4334-bfac-6b16bafd1e80], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/7ccbe92f-0d4c-4080-b762-639874649785], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/816bb0d9-b0c8-4d65-91e9-7778b909942a], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/5b7a933a-99bc-4e80-a656-77e69b28d067], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/1da795c0-68e2-4433-bf75-edccf1e2e52b], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/f6228c58-2a49-4cfd-b48c-d7774aaa3c92], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/01c7ebe5-2444-41d5-8fa1-7fc2cf8dd1ff], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/46199d2a-269b-469e-9fef-6ae8b7e19bfa], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/1fd032b4-eb67-479e-a1a5-70d11afaf834], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/66990d16-ee90-40f9-9747-f08a36dde016], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/8e05d0e0-8669-43cf-906e-7dc81ff887c3], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/a929bdfc-6663-413b-bead-5dca8faf48b2], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/9b36aa2a-e949-46b7-aeaa-d30534f43a42], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/cdb4c0ea-93aa-4e11-ad75-97bd2aff7204], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/af4c54b8-976b-457a-a8b1-d2fd99888983], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/10403295-3594-4e69-841f-2a472a49725e], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/89e731f0-4214-4fab-b11e-81522896b8b1], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/ddacf50d-9b5a-48dc-8100-58623baf46bf], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/ee8b3503-4970-4fc9-84c6-6ade074c2091], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/3dc1e0cf-7c3b-488f-9e51-1d9f8c351ec9], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/d8ccc1b1-2cc4-4562-8901-f3f609ea56a4], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/d81e932b-5ff7-46bb-bb43-02683595a308], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/62f10d11-6f09-459b-aaf1-744a47e12cee], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/48e50b2a-3c03-4de1-88fd-f309eddb3406], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/88f6f264-c2f8-470a-aced-3b71e5841f9f], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/56050f58-75f6-476a-87ec-778d8288470d], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/87af625c-3db8-4217-bf75-4f0ba77a8e8c], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/bc7718de-5a3a-4ba5-a287-0d53e467bb56], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/c733defb-b26c-4150-83b4-926aef54a680], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/7a139d70-c0cf-43e8-bb4c-67a8751cfb30], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/56e82027-53e0-4ae3-b801-8af60dfb85ca], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/b613849c-baf6-472e-8a60-c12042d590ef], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/8daaa0a6-6898-429f-9bef-a01eb85e9cc3], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/f5b22a70-a674-4360-91b8-35835e42a899], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/5d3775fb-b086-4663-afbe-4091541e872d], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/97c32c53-7d5d-40d2-8578-fb4d46c7510e], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/3464d06e-b807-4371-9210-056bbcfc0056], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/b35c0233-06e7-420e-a529-78bd61932845], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/38847aba-cbff-4e04-8afc-f79fe835b3dd], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/c4f24f4d-f20b-4a3b-94b1-88a1044ff94f], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/5b1a4567-33c3-494d-8c5b-120eb87c4525], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/0289e13b-0f71-4142-9893-0733d8a0a8a9], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/dce2f1d5-67ea-4e59-af84-57e1360aa303], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/2ae3d893-9dcd-4b49-a71c-634be9aa27ab], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/de960b0c-fb2c-46ce-9c7c-23c4e9418a1d], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/40100d37-4019-465a-ae47-c9bd29b461b8], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/362aaa05-cede-486b-b95a-c34141add768], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/3eb39776-3160-44f7-9c88-fffca32283f6], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/bb3f4d2a-8d95-4efb-8460-66a5a6616f71], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/d5627c87-3f78-42e0-b8b8-13f406e08dba], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/efe2a0ac-6bf2-4667-8412-ac15b1b55acc], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/d27e2d9b-40d7-49ea-9a18-5bdb7569d205], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/a2994f26-f282-4019-9d08-868af7ec0bee], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/cfef34eb-4b01-44fa-91e4-29498bde50be], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/4f63a4b9-05b0-44be-8cd8-7a500f5a4ce0], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/951d6708-77ac-4672-9d8f-1e5660dbc74f], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/d12f5f12-9fdf-48e7-9ba0-5f7b511cd91b], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/0a1d9b37-9ea5-4e19-afad-b88968a7c5e0], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/0a9ee26d-9b53-48a5-8489-a5e70fded241], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/0c3c6e70-3546-42d7-82d6-23520b3a1f03], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/cf0088f2-5b1e-440f-a596-75d5e4fa267a], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/d9c3fb2d-7dfd-407c-9366-f57a0bbcf209], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/abc5cd5c-ca5b-4ce6-b9f7-51c6c85ece05], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/935e3bfa-c56f-4062-ad9b-ee550e380bd8], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/1728cc55-2741-42b8-b710-db09cfbed8d4], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/ec9b3ec1-9db5-42f1-874f-da0d7df81087], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/9bf7b2c3-7144-4bbb-a3bc-ece087f2281b], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/0b795c18-99b4-4bb8-9ede-795d14c58bec], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/fc39c291-cca1-402a-9c37-4a8a4df9ac68], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/2cec393a-1a19-479b-bed5-336c561a572b], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/0ac03cf4-8cf2-47ec-a1c5-36aeaf7e6572], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/d7b555aa-5a49-4cba-8499-6611ff2f7791], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/e1764b4e-ab58-4f19-8d2a-88b7fd98a71f], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/5932a5e1-5ef8-49ff-96a0-a6326e569442], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/e60af8f9-ed4a-459d-aa79-5c697bad107e], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/a01c7d15-0ee6-4c5c-a787-062921e25585], RequestData [method=GET, url=https://test.site.gov/Attachments/Order/ceb1c6e8-fdaf-45e3-92f1-021cef03d940], RequestData [method=GET, url=https://test.site.gov/MyDms], RequestData [method=GET, url=https://test.site.gov/App], RequestData [method=GET, url=http://scetvradio.vo.llnwd.net/o33/EdDiv/PSC_DMS/Final/DMSandEFile_Menu_Jan9/index.html], RequestData [method=GET, url=https://test.site.gov/Home/Privacy], RequestData [method=GET, url=https://test.site.gov/favicon.ico], RequestData [method=GET, url=https://test.site.gov/Content/css?v=i0UGyJoiOf-p7RmDeKgpJ8_MEBnwSJQA_0u-9mobZA41]] 2017-11-16 17:27:46,112 [Crawler-20171116172730-1-1] DEBUG Finished https://test.site.gov/Web/Orders 2017-11-16 17:27:49,999 [CoreLib-TimeoutManager] DEBUG Closing expired connections 2017-11-16 17:27:49,999 [CoreLib-TimeoutManager] DEBUG Closing connections idle longer than 60000 MILLISECONDS 2017-11-16 17:27:54,473 [IndexUpdater] DEBUG Processing documents in IndexUpdater queue. 2017-11-16 17:27:54,473 [IndexUpdater] DEBUG Getting documents in IndexUpdater queue. 2017-11-16 17:27:54,488 [IndexUpdater] INFO Processing 1/1 docs (Doc:{access 15ms}, Mem:{used 161MB, heap 224MB, max 494MB}) 2017-11-16 17:27:54,488 [IndexUpdater] DEBUG Indexing https://test.site.gov/Web/Orders 2017-11-16 17:27:54,542 [IndexUpdater] DEBUG Skipped. This document is not a index target. 2017-11-16 17:27:54,573 [IndexUpdater] DEBUG Updated 1 access results. The execution time is 31ms. 2017-11-16 17:27:54,573 [IndexUpdater] DEBUG Getting documents in IndexUpdater queue. 2017-11-16 17:27:54,573 [IndexUpdater] INFO Processing no docs (Doc:{access 0ms, cleanup 31ms}, Mem:{used 167MB, heap 224MB, max 494MB}) 2017-11-16 17:27:54,573 [IndexUpdater] DEBUG Processed documents in IndexUpdater queue. 2017-11-16 17:27:55,037 [CoreLib-TimeoutManager] DEBUG Closing expired connections 2017-11-16 17:27:55,037 [CoreLib-TimeoutManager] DEBUG Closing connections idle longer than 60000 MILLISECONDS 2017-11-16 17:27:56,121 [Crawler-20171116172730-1-1] DEBUG The url is null. (0) 2017-11-16 17:28:00,068 [CoreLib-TimeoutManager] DEBUG Closing expired connections 2017-11-16 17:28:00,068 [CoreLib-TimeoutManager] DEBUG Closing connections idle longer than 60000 MILLISECONDS 2017-11-16 17:28:04,473 [IndexUpdater] DEBUG Processing documents in IndexUpdater queue. 2017-11-16 17:28:04,473 [IndexUpdater] DEBUG Getting documents in IndexUpdater queue. 2017-11-16 17:28:04,475 [IndexUpdater] INFO Processing no docs (Doc:{access 2ms, cleanup 31ms}, Mem:{used 167MB, heap 224MB, max 494MB}) 2017-11-16 17:28:04,475 [IndexUpdater] DEBUG Processed documents in IndexUpdater queue. 2017-11-16 17:28:05,098 [CoreLib-TimeoutManager] DEBUG Closing expired connections 2017-11-16 17:28:05,098 [CoreLib-TimeoutManager] DEBUG Closing connections idle longer than 60000 MILLISECONDS 2017-11-16 17:28:06,629 [Crawler-20171116172730-1-1] DEBUG The url is null. (1) 2017-11-16 17:28:10,098 [CoreLib-TimeoutManager] DEBUG Closing expired connections 2017-11-16 17:28:10,098 [CoreLib-TimeoutManager] DEBUG Closing connections idle longer than 60000 MILLISECONDS 2017-11-16 17:28:14,474 [IndexUpdater] DEBUG Processing documents in IndexUpdater queue. 2017-11-16 17:28:14,474 [IndexUpdater] DEBUG Getting documents in IndexUpdater queue. 2017-11-16 17:28:14,476 [IndexUpdater] INFO Processing no docs (Doc:{access 2ms, cleanup 31ms}, Mem:{used 167MB, heap 224MB, max 494MB}) 2017-11-16 17:28:14,477 [IndexUpdater] DEBUG Processed documents in IndexUpdater queue. 2017-11-16 17:28:15,099 [CoreLib-TimeoutManager] DEBUG Closing expired connections 2017-11-16 17:28:15,099 [CoreLib-TimeoutManager] DEBUG Closing connections idle longer than 60000 MILLISECONDS 2017-11-16 17:28:17,131 [Crawler-20171116172730-1-1] DEBUG The url is null. (2) 2017-11-16 17:28:20,100 [CoreLib-TimeoutManager] DEBUG Closing expired connections 2017-11-16 17:28:20,100 [CoreLib-TimeoutManager] DEBUG Closing connections idle longer than 60000 MILLISECONDS 2017-11-16 17:28:24,474 [IndexUpdater] DEBUG Processing documents in IndexUpdater queue. 2017-11-16 17:28:24,474 [IndexUpdater] DEBUG Getting documents in IndexUpdater queue. 2017-11-16 17:28:24,474 [IndexUpdater] INFO Processing no docs (Doc:{access 0ms, cleanup 31ms}, Mem:{used 167MB, heap 224MB, max 494MB}) 2017-11-16 17:28:24,474 [IndexUpdater] DEBUG Processed documents in IndexUpdater queue. 2017-11-16 17:28:25,101 [CoreLib-TimeoutManager] DEBUG Closing expired connections 2017-11-16 17:28:25,101 [CoreLib-TimeoutManager] DEBUG Closing connections idle longer than 60000 MILLISECONDS 2017-11-16 17:28:27,647 [Crawler-20171116172730-1-1] DEBUG The url is null. (3) 2017-11-16 17:28:30,124 [CoreLib-TimeoutManager] DEBUG Closing expired connections 2017-11-16 17:28:30,124 [CoreLib-TimeoutManager] DEBUG Closing connections idle longer than 60000 MILLISECONDS 2017-11-16 17:28:34,485 [IndexUpdater] DEBUG Processing documents in IndexUpdater queue. 2017-11-16 17:28:34,485 [IndexUpdater] DEBUG Getting documents in IndexUpdater queue. 2017-11-16 17:28:34,488 [IndexUpdater] INFO Processing no docs (Doc:{access 3ms, cleanup 31ms}, Mem:{used 167MB, heap 224MB, max 494MB}) 2017-11-16 17:28:34,488 [IndexUpdater] DEBUG Processed documents in IndexUpdater queue. 2017-11-16 17:28:35,139 [CoreLib-TimeoutManager] DEBUG Closing expired connections 2017-11-16 17:28:35,139 [CoreLib-TimeoutManager] DEBUG Closing connections idle longer than 60000 MILLISECONDS 2017-11-16 17:28:38,162 [Crawler-20171116172730-1-1] DEBUG The url is null. (4) 2017-11-16 17:28:40,139 [CoreLib-TimeoutManager] DEBUG Closing expired connections 2017-11-16 17:28:40,139 [CoreLib-TimeoutManager] DEBUG Closing connections idle longer than 60000 MILLISECONDS 2017-11-16 17:28:44,485 [IndexUpdater] DEBUG Processing documents in IndexUpdater queue. 2017-11-16 17:28:44,485 [IndexUpdater] DEBUG Getting documents in IndexUpdater queue. 2017-11-16 17:28:44,493 [IndexUpdater] INFO Processing no docs (Doc:{access 8ms, cleanup 31ms}, Mem:{used 125MB, heap 224MB, max 494MB}) 2017-11-16 17:28:44,493 [IndexUpdater] DEBUG Processed documents in IndexUpdater queue. 2017-11-16 17:28:44,507 [CoreLib-TimeoutManager] INFO [SYSTEM MONITOR] {"os":{"memory":{"physical":{"free":3898707968,"total":12883316736},"swap_space":{"free":11201110016,"total":28989444096}},"cpu":{"percent":0},"load_averages":null},"process":{"file_descriptor":{"open":-1,"max":-1},"cpu":{"percent":12,"total":27703},"virtual_memory":{"total":534474752}},"jvm":{"memory":{"heap":{"used":176322464,"committed":235139072,"max":518979584,"percent":33},"non_heap":{"used":91116456,"committed":95719424}},"pools":{"direct":{"count":40,"used":85983265,"capacity":85983264},"mapped":{"count":0,"used":0,"capacity":0}},"gc":{"young":{"count":14,"time":125},"old":{"count":3,"time":54}},"threads":{"count":57,"peak":57},"classes":{"loaded":11256,"total_loaded":11261,"unloaded":5},"uptime":73639},"elasticsearch":{"nodes":{"48_IXnOZTsGziaBau7tg8w":{"timestamp":1510871324416,"name":"Node 1","transport_address":"127.0.0.1:9301","host":"127.0.0.1","ip":"127.0.0.1:9301","roles":["master","data","ingest"],"indices":{"docs":{"count":219,"deleted":6},"store":{"size_in_bytes":588541,"throttle_time_in_millis":0},"indexing":{"index_total":68,"index_time_in_millis":171,"index_current":0,"index_failed":0,"delete_total":19,"delete_time_in_millis":9,"delete_current":0,"noop_update_total":0,"is_throttled":false,"throttle_time_in_millis":0},"get":{"total":24,"time_in_millis":11,"exists_total":14,"exists_time_in_millis":11,"missing_total":10,"missing_time_in_millis":0,"current":0},"search":{"open_contexts":20,"query_total":2928,"query_time_in_millis":1229,"query_current":0,"fetch_total":536,"fetch_time_in_millis":157,"fetch_current":0,"scroll_total":522,"scroll_time_in_millis":46489562,"scroll_current":20,"suggest_total":0,"suggest_time_in_millis":0,"suggest_current":0},"merges":{"current":0,"current_docs":0,"current_size_in_bytes":0,"total":1,"total_time_in_millis":150,"total_docs":62,"total_size_in_bytes":71049,"total_stopped_time_in_millis":0,"total_throttled_time_in_millis":0,"total_auto_throttle_in_bytes":1195376640},"refresh":{"total":570,"total_time_in_millis":959,"listeners":0},"flush":{"total":236,"total_time_in_millis":26146},"warmer":{"current":0,"total":361,"total_time_in_millis":23},"query_cache":{"memory_size_in_bytes":0,"total_count":0,"hit_count":0,"miss_count":0,"cache_size":0,"cache_count":0,"evictions":0},"fielddata":{"memory_size_in_bytes":8784,"evictions":0},"completion":{"size_in_bytes":0},"segments":{"count":92,"memory_in_bytes":248785,"terms_memory_in_bytes":202899,"stored_fields_memory_in_bytes":28704,"term_vectors_memory_in_bytes":0,"norms_memory_in_bytes":768,"points_memory_in_bytes":430,"doc_values_memory_in_bytes":15984,"index_writer_memory_in_bytes":0,"version_map_memory_in_bytes":361,"fixed_bit_set_memory_in_bytes":0,"max_unsafe_auto_id_timestamp":1510868407562,"file_sizes":{}},"translog":{"operations":14,"size_in_bytes":441427},"request_cache":{"memory_size_in_bytes":10885,"evictions":0,"hit_count":0,"miss_count":40},"recovery":{"current_as_source":0,"current_as_target":0,"throttle_time_in_millis":0}},"os":{"timestamp":1510871324421,"cpu":{"percent":8},"mem":{"total_in_bytes":12883316736,"free_in_bytes":3898388480,"used_in_bytes":8984928256,"free_percent":30,"used_percent":70},"swap":{"total_in_bytes":28989444096,"free_in_bytes":11203006464,"used_in_bytes":17786437632}},"process":{"timestamp":1510871324447,"open_file_descriptors":-1,"max_file_descriptors":-1,"cpu":{"percent":0,"total_in_millis":132687},"mem":{"total_virtual_in_bytes":1363505152}},"jvm":{"timestamp":1510871324447,"uptime_in_millis":2805421,"mem":{"heap_used_in_bytes":566523824,"heap_used_percent":54,"heap_committed_in_bytes":796393472,"heap_max_in_bytes":1037959168,"non_heap_used_in_bytes":203793920,"non_heap_committed_in_bytes":217698304,"pools":{"young":{"used_in_bytes":26942392,"max_in_bytes":286326784,"peak_used_in_bytes":71630848,"peak_max_in_bytes":286326784},"survivor":{"used_in_bytes":5745696,"max_in_bytes":35782656,"peak_used_in_bytes":8912896,"peak_max_in_bytes":35782656},"old":{"used_in_bytes":533835736,"max_in_bytes":715849728,"peak_used_in_bytes":563787848,"peak_max_in_bytes":715849728}}},"threads":{"count":122,"peak_count":134},"gc":{"collectors":{"young":{"collection_count":118,"collection_time_in_millis":1073},"old":{"collection_count":7,"collection_time_in_millis":218}}},"buffer_pools":{"direct":{"count":100,"used_in_bytes":174488316,"total_capacity_in_bytes":174488315},"mapped":{"count":100,"used_in_bytes":476477,"total_capacity_in_bytes":476477}},"classes":{"current_loaded_count":21898,"total_loaded_count":22009,"total_unloaded_count":111}},"thread_pool":{"bulk":{"threads":8,"queue":0,"active":0,"rejected":0,"largest":8,"completed":60},"fetch_shard_started":{"threads":1,"queue":0,"active":0,"rejected":0,"largest":16,"completed":58},"fetch_shard_store":{"threads":0,"queue":0,"active":0,"rejected":0,"largest":0,"completed":0},"flush":{"threads":4,"queue":0,"active":0,"rejected":0,"largest":4,"completed":301},"force_merge":{"threads":0,"queue":0,"active":0,"rejected":0,"largest":0,"completed":0},"generic":{"threads":4,"queue":0,"active":0,"rejected":0,"largest":4,"completed":342},"get":{"threads":8,"queue":0,"active":0,"rejected":0,"largest":8,"completed":16},"index":{"threads":8,"queue":0,"active":0,"rejected":0,"largest":8,"completed":948},"listener":{"threads":0,"queue":0,"active":0,"rejected":0,"largest":0,"completed":0},"management":{"threads":4,"queue":0,"active":1,"rejected":0,"largest":4,"completed":494},"refresh":{"threads":4,"queue":0,"active":0,"rejected":0,"largest":4,"completed":28110},"search":{"threads":13,"queue":0,"active":0,"rejected":0,"largest":13,"completed":4410},"snapshot":{"threads":0,"queue":0,"active":0,"rejected":0,"largest":0,"completed":0},"warmer":{"threads":0,"queue":0,"active":0,"rejected":0,"largest":0,"completed":0}},"fs":{"timestamp":1510871324447,"total":{"total_in_bytes":1023681753088,"free_in_bytes":638920826880,"available_in_bytes":638920826880},"data":[{"path":"C:\Users\jmcclure\Downloads\fess-11.4.3\fess-11.4.3\es\data\node_1\nodes\0","mount":"(C:)","type":"NTFS","total_in_bytes":1023681753088,"free_in_bytes":638920826880,"available_in_bytes":638920826880}]},"transport":{"server_open":26,"rx_count":1842,"rx_size_in_bytes":1055704,"tx_count":1841,"tx_size_in_bytes":2157878}}}},"timestamp":1510871324507} 2017-11-16 17:28:45,508 [CoreLib-TimeoutManager] DEBUG Closing expired connections 2017-11-16 17:28:45,508 [CoreLib-TimeoutManager] DEBUG Closing connections idle longer than 60000 MILLISECONDS 2017-11-16 17:28:45,508 [CoreLib-TimeoutManager] DEBUG http-outgoing-0: Close connection 2017-11-16 17:28:48,665 [Crawler-20171116172730-1-1] DEBUG The url is null. (5) 2017-11-16 17:28:50,510 [CoreLib-TimeoutManager] DEBUG Closing expired connections 2017-11-16 17:28:50,510 [CoreLib-TimeoutManager] DEBUG Closing connections idle longer than 60000 MILLISECONDS 2017-11-16 17:28:54,489 [IndexUpdater] DEBUG Processing documents in IndexUpdater queue. 2017-11-16 17:28:54,489 [IndexUpdater] DEBUG Getting documents in IndexUpdater queue. 2017-11-16 17:28:54,492 [IndexUpdater] INFO Processing no docs (Doc:{access 3ms, cleanup 31ms}, Mem:{used 127MB, heap 224MB, max 494MB}) 2017-11-16 17:28:54,492 [IndexUpdater] DEBUG Processed documents in IndexUpdater queue. 2017-11-16 17:28:55,510 [CoreLib-TimeoutManager] DEBUG Closing expired connections 2017-11-16 17:28:55,510 [CoreLib-TimeoutManager] DEBUG Closing connections idle longer than 60000 MILLISECONDS

marevol commented 6 years ago

Try the following setting:

URLs:
https://test.site.gov/Web/Orders/

Include for crawling:
https://test.site.gov/Web/Orders/.*

Include for indexing:
https://test.site.gov/Attachments/Order/.*
zaryk commented 6 years ago

Looks like it has the same results.

I get the logs to show they are downloading the pdf and extracting if I blank out the Include for crawling and set include for indexing as "**https://test.site.gov/Attachments/Order/***" Still not searchable. The pdf urls are are extensionless. But, I may end up using the FileSystem method due to an issue I saw with how the page is presented. The pdfs are in a jquery grid with paging.

EDIT: Still would like to figure this out just for future reference.

marevol commented 6 years ago

The pattern is https://test.site.gov/Web/Orders/.*, not https://test.site.gov/Web/Orders/* It's Java Regular Expression.