search

hellemo / ShortCodes.jl

Short codes to embed content in Pluto notebooks
MIT License
49 stars 4 forks source link

HTTP 500 Error with DOI #11

Closed stefanbringuier closed 1 year ago

stefanbringuier commented 1 year ago

I'm noticing issues since the bump in HTTP version in v0.3.3 commit d1a499135da73fb608528bfe4cbcdf71510802b7. The response times out due to:


HTTP.Exceptions.StatusError(500, "GET", "/index/api/v1/metadata/10.1186/1756-8722-6-59", HTTP.Messages.Response:

"""
HTTP/1.1 500 Internal Server Error
Content-Type: application/json
Set-Cookie: webpy_session_id=502faf465e77c8c7eae682df0304f4716085fd6c; Path=/; httponly
Content-Length: 43
Date: Thu, 02 Feb 2023 18:59:12 GMT
Server: lighttpd/1.4.53
HTTP status code 500: Internal Server Error"""

)

However, I'm suspecting its not HTTP.jl but changes to the opencitations REST API ? Currently https://github.com/hellemo/ShortCodes.jl/blob/b0e8bf208338195c77b3de9c76e7c34ffc7f1aaa/src/doi.jl#L52

is using metadata in the link, but it looks like it may need to be references, I'm not sure. Thanks for the useful package!

hellemo commented 1 year ago

Thanks for reporting. It should be the right one according to the documentation: https://opencitations.net/index/coci/api/v1#/metadata/{dois} I'm guessing it's a server issue, will check back in a couple of days.

stefanbringuier commented 1 year ago

@hellemo I'm still experiencing this issue, is it the same on your end?

hellemo commented 1 year ago

I still have the same problem. I checked with opencitations, and they will look into it. They say it only happens occationally, but for me it is very consistent since you raised the issue.

AshtonSBradley commented 1 year ago

shortcodes DOI gives me this

Failed to show value:

HTTP.Exceptions.StatusError(500, "GET", "/index/api/v1/metadata/10.1137/141000671", HTTP.Messages.Response:

"""

HTTP/1.1 500 Internal Server Error

Content-Type: application/json

Set-Cookie: webpy_session_id=1729cd4b01e1be0b30683b759306df1dadca9775; Path=/; httponly

Content-Length: 43

Date: Thu, 16 Feb 2023 21:31:37 GMT

Server: lighttpd/1.4.53

HTTP status code 500: Internal Server Error""")
AshtonSBradley commented 1 year ago

I still have the same problem. I checked with opencitations, and they will look into it. They say it only happens occationally, but for me it is very consistent since you raised the issue.

yeah, it seems to be permanent for DOI. For twitter it is intermittent, but frequent.

AshtonSBradley commented 1 year ago

I still have the same problem. I checked with opencitations, and they will look into it. They say it only happens occationally, but for me it is very consistent since you raised the issue.

do they have an issue open for it?

hellemo commented 1 year ago

do they have an issue open for it?

This looks like a relevant issue: https://github.com/opencitations/api/issues/7

AshtonSBradley commented 1 year ago

this is a real bummer... so much that now depends on ShortCodes... how can I help fix this?

AshtonSBradley commented 1 year ago

Maybe i misunderstand the issue, but can I suggest rolling back the HTTP version bump for ShortCodes, https://github.com/hellemo/ShortCodes.jl/commit/d1a499135da73fb608528bfe4cbcdf71510802b7, until https://github.com/opencitations/api/issues/7 is fixed?

stefanbringuier commented 1 year ago

this is a real bummer... so much that now depends on ShortCodes... how can I help fix this?

This is indeed unfortunate. I've tried HTTP.get calls to a few of the different REST API flavors; all time out. I'm wondering if obtaining a API access token would solve this; although its not required.

Maybe i misunderstand the issue, but can I suggest rolling back the HTTP version bump for ShortCodes, d1a4991, until opencitations/api#7 is fixed?

I don't think this was the issue but you could test it out by creating a Julia project environment:

julia>]
(@v1.8) pkg> activate --temp
(@v1.8) pkg> add ShortCodes@0.3.3
AshtonSBradley commented 1 year ago

thanks for the info, I will try it with a token

AshtonSBradley commented 1 year ago

bit of a newb question, but any advice as to how to insert a token to ShortCodes API calls?

stefanbringuier commented 1 year ago

bit of a newb question, but any advice as to how to insert a token to ShortCodes API calls?

As far as I can see in doi.jl there are no functions that support receiving the header (i.e token) keyword for HTTP.get. Therefore, I think what you would need to do is import ShortCodes and add methods that receive two variables with the second being a header/token input for the following functions:

https://github.com/hellemo/ShortCodes.jl/blob/b0e8bf208338195c77b3de9c76e7c34ffc7f1aaa/src/doi.jl#L51 https://github.com/hellemo/ShortCodes.jl/blob/b0e8bf208338195c77b3de9c76e7c34ffc7f1aaa/src/doi.jl#L63 https://github.com/hellemo/ShortCodes.jl/blob/b0e8bf208338195c77b3de9c76e7c34ffc7f1aaa/src/doi.jl#L72

You could also just fork and modify the functions to take a keyword instead.

As a follow-up I tried out my suggestion for using v0.3.3 and I still was getting issues.

AshtonSBradley commented 1 year ago

This points to a new API going forward. Do you see any issues with moving ShortCodes to this?

https://github.com/opencitations/api/issues/7#issuecomment-1455224146

hellemo commented 1 year ago

Thanks, moving to the new API should be doable, initial work here: https://github.com/hellemo/ShortCodes.jl/pull/12 if anyone wants to try it out. Apart from testing, some more work will be needed to add the citation counts back, they are not included in the new metadata API call.

AshtonSBradley commented 1 year ago

I think this should reopen as papers with no citations error at present:

using ShortCodes DOI("10.1103/PhysRev.65.117")

Failed to show value:

BoundsError: attempt to access 0-element JSON3.Array{Union{}, Vector{UInt8}, Vector{UInt64}} at index [1]

throw_boundserror(::JSON3.Array{Union{}, Vector{UInt8}, Vector{UInt64}}, ::Tuple{Int64})@abstractarray.jl:703 checkbounds@abstractarray.jl:668[inlined] getindex(::JSON3.Array{Union{}, Vector{UInt8}, Vector{UInt64}}, ::Int64)@JSON3.jl:158 var"##fetch_metadata_unmemoized"(::String)@doi.jl:68

5@Memoize.jl:62[inlined]

get!@iddict.jl:178[inlined] fetch_metadata@Memoize.jl:61[inlined] var"##fetch_metadata_unmemoized"(::ShortCodes.DOI{String})@doi.jl:63

3@Memoize.jl:62[inlined]

get!(::ShortCodes.var"#3#4"{ShortCodes.DOI{String}}, ::IdDict{Any, Any}, ::Any)@iddict.jl:178 fetch_metadata(::ShortCodes.DOI{String})@Memoize.jl:61 getproperty(::ShortCodes.DOI{String}, ::Symbol)@doi.jl:26 emph_author(::ShortCodes.DOI{String})@doi.jl:104 show(::IOContext{IOBuffer}, ::MIME{Symbol("text/html")}, ::ShortCodes.DOI{String})@doi.jl:43 show_richest(::IOContext{IOBuffer}, ::Any)@PlutoRunner.jl:1186 show_richest_withreturned@PlutoRunner.jl:1100[inlined] format_output_default(::Any, ::Any)@PlutoRunner.jl:1000

format_output#61@PlutoRunner.jl:1017[inlined]

53@PlutoRunner.jl:909[inlined]

var"#with_io_to_logs#115"(::Bool, ::Base.CoreLogging.LogLevel, ::typeof(Main.PlutoRunner.with_io_to_logs), ::Main.PlutoRunner.var"#53#56"{Module, ShortCodes.DOI{String}, Dict{Tuple{UInt64, Int64}, Int64}})@PlutoRunner.jl:2424 (::Main.PlutoRunner.var"#119#120"{Bool, Base.CoreLogging.LogLevel, Main.PlutoRunner.var"#53#56"{Module, ShortCodes.DOI{String}, Dict{Tuple{UInt64, Int64}, Int64}}})()@PlutoRunner.jl:2448 with_logstate(::Function, ::Any)@logging.jl:511 with_logger@logging.jl:623[inlined]

with_logger_and_io_to_logs#118@PlutoRunner.jl:2447[inlined]

var"#formatted_result_of#50"(::Bool, ::typeof(Main.PlutoRunner.formatted_result_of), ::Base.UUID, ::Base.UUID, ::Bool, ::Vector{String}, ::Nothing, ::Module)@PlutoRunner.jl:908 top-level scope@WorkspaceManager.jl:527

DOI("10.1103/PhysRevA.107.010101")

Failed to show value:

BoundsError: attempt to access 0-element JSON3.Array{Union{}, Vector{UInt8}, Vector{UInt64}} at index [1]

throw_boundserror(::JSON3.Array{Union{}, Vector{UInt8}, Vector{UInt64}}, ::Tuple{Int64})@abstractarray.jl:703 checkbounds@abstractarray.jl:668[inlined] getindex(::JSON3.Array{Union{}, Vector{UInt8}, Vector{UInt64}}, ::Int64)@JSON3.jl:158 var"##fetch_metadata_unmemoized"(::String)@doi.jl:68

5@Memoize.jl:62[inlined]

get!@iddict.jl:178[inlined] fetch_metadata@Memoize.jl:61[inlined] var"##fetch_metadata_unmemoized"(::ShortCodes.DOI{String})@doi.jl:63

3@Memoize.jl:62[inlined]

get!(::ShortCodes.var"#3#4"{ShortCodes.DOI{String}}, ::IdDict{Any, Any}, ::Any)@iddict.jl:178 fetch_metadata(::ShortCodes.DOI{String})@Memoize.jl:61 getproperty(::ShortCodes.DOI{String}, ::Symbol)@doi.jl:26 emph_author(::ShortCodes.DOI{String})@doi.jl:104 show(::IOContext{IOBuffer}, ::MIME{Symbol("text/html")}, ::ShortCodes.DOI{String})@doi.jl:43 show_richest(::IOContext{IOBuffer}, ::Any)@PlutoRunner.jl:1186 show_richest_withreturned@PlutoRunner.jl:1100[inlined] format_output_default(::Any, ::Any)@PlutoRunner.jl:1000

format_output#61@PlutoRunner.jl:1017[inlined]

53@PlutoRunner.jl:909[inlined]

var"#with_io_to_logs#115"(::Bool, ::Base.CoreLogging.LogLevel, ::typeof(Main.PlutoRunner.with_io_to_logs), ::Main.PlutoRunner.var"#53#56"{Module, ShortCodes.DOI{String}, Dict{Tuple{UInt64, Int64}, Int64}})@PlutoRunner.jl:2424 (::Main.PlutoRunner.var"#119#120"{Bool, Base.CoreLogging.LogLevel, Main.PlutoRunner.var"#53#56"{Module, ShortCodes.DOI{String}, Dict{Tuple{UInt64, Int64}, Int64}}})()@PlutoRunner.jl:2448 with_logstate(::Function, ::Any)@logging.jl:511 with_logger@logging.jl:623[inlined]

with_logger_and_io_to_logs#118@PlutoRunner.jl:2447[inlined]

var"#formatted_result_of#50"(::Bool, ::typeof(Main.PlutoRunner.formatted_result_of), ::Base.UUID, ::Base.UUID, ::Bool, ::Vector{String}, ::Nothing, ::Module)@PlutoRunner.jl:908 top-level scope@WorkspaceManager.jl:527