HTTP 404 on generated site

yaakov-h commented 7 years ago

Hi,

I've been using Microsoft.SourceBrowser from NuGet, but it doesn't support the new project format with MSBuild 15.0, or new syntax with Roslyn 2.0.

I've managed to build the latest code from source.

I used the following command-line to generate the index and website:

C:\Git\SourceBrowser\src\HtmlGenerator\bin\Debug\net46\HtmlGenerator.exe  /force /out:__SourceBrowser__ /in:ProjectList.txt

The generated site appears to mostly work, but there's no main page. URLs such as /i.txt work, but / just returns a HTTP 404.

At first I thought this might be an IIS issue, so I used dotnet to run Kestrel and got the same result.

C:\CW\__SourceBrowser__>dotnet .\Microsoft.SourceBrowser.SourceIndexServer.dll --help
Hosting environment: Production
Content root path: C:\CW\__SourceBrowser__
Now listening on: http://localhost:5000
Application started. Press Ctrl+C to shut down.
info: Microsoft.AspNetCore.Hosting.Internal.WebHost[1]
      Request starting HTTP/1.1 GET http://localhost:5000/
info: Microsoft.AspNetCore.Hosting.Internal.WebHost[2]
      Request finished in 170.0031ms 404
info: Microsoft.AspNetCore.Hosting.Internal.WebHost[1]
      Request starting HTTP/1.1 GET http://localhost:5000/
info: Microsoft.AspNetCore.Hosting.Internal.WebHost[2]
      Request finished in 0.7488ms 404

Am I missing a step, or is the generated site actually incomplete?

KirillOsenkov commented 7 years ago

Yes, you need to actually host the generated site in IIS (unfortunately). Read more here: https://github.com/KirillOsenkov/SourceBrowser#conceptual-design

The website is in the SourceIndexServer project. You need to deploy that project somewhere (like an Azure website, or an IIS server on your own machine).

yaakov-h commented 7 years ago

I got the 404 in IIS too, and only resorted to dotnet to try debug what was going on.

I thought I deploy the /out: path? It has web.config and the whole index... just not the main HTML page, it appears.

KirillOsenkov commented 7 years ago

Is there an index.html in the HtmlGenerator output directory? Have you tried setting the SourceIndexServer project as startup in VS and hitting F5?

KirillOsenkov commented 7 years ago

Hi, I think I've fixed the issues you were seeing. SDK-style projects should now be supported as well. Could you please try again and let me know if it's still not working for you? Thanks!

KirillOsenkov commented 7 years ago

The structure of output has changed. It now has the root-level \Index folder which has the ASP.NET Core app, and the actual index is in another Index subfolder (\Index\Index). I know it's a bit confusing but it seems to be working for us.

@alexperovich

yaakov-h commented 7 years ago

Thanks, I'll try again soon.

yaakov-h commented 7 years ago

@KirillOsenkov I've tried again with latest code from this repo (master branch), with no such luck.

Here's my setup:

sourcebrowser_iis

In C:\cw\__SourceBrowser__, there are no HTML files. It appears to just be the ASP.NET Core app, plus web.config.

In C:\cw\__SourceBrowser__\Index, there is the index, plus the following files:

C:\CW\__SourceBrowser__\index>dir /A:-D
 Volume in drive C has no label.
 Volume Serial Number is [REDACTED]

 Directory of C:\CW\__SourceBrowser__\index

27/09/2017  10:02 AM               144 0000000000.html
29/08/2017  10:11 AM               156 AffiliateLinks.txt
27/09/2017  10:02 AM            25,094 Assemblies.txt
27/09/2017  10:02 AM        11,355,415 DeclaredSymbols.txt
27/09/2017  10:01 AM            26,033 Errors.txt
27/09/2017  10:02 AM             9,725 Huffman.txt
27/09/2017  10:02 AM               156 i.txt
27/09/2017  10:01 AM            22,201 ProcessedAssemblies.txt
27/09/2017  10:02 AM            20,866 Projects.txt
27/09/2017  10:02 AM               455 results.html
27/09/2017  10:01 AM               566 SolutionExplorer.html
27/09/2017  10:01 AM             3,731 TopReferencedAssemblies.txt
              12 File(s)     11,464,542 bytes
               0 Dir(s)  631,681,859,584 bytes free

If I copy the contents of src\SourceIndexServer\wwwroot into C:\CW\__SourceBrowser__\index\ then I can load each file individually in the web browser, but just not the root of the site.

If I go to http://localhost:2080/index.html I can see the source browser for just a moment, before the javascript redirects to http://localhost:2080/ which returns a 404.

I'd think something is broken in ASP.NET Core's UseDefaultPage() machinery.

KirillOsenkov commented 7 years ago

Can you paste what's inside that Errors.txt? There were a bunch of exceptions it looks like.

KirillOsenkov commented 7 years ago

Also, could you run HtmlGenerator on the built-in TestSolution? Just set HtmlGenerator as startup, in Project properties select Debug -> Application arguments: ..\..\TestCode\TestSolution.sln /out:..\SourceIndexServer\index and run. It should generate a website. You can then run SourceIndexServer from that website and make sure it runs.

yaakov-h commented 7 years ago

Errors are below, redacted and deduplicated. First 3 are somewhat expected.

A project with assembly name [NAME] was already generated, skipping current project: [PATH]

Workspace failed: Project file not found: '[PATH]'

Workspace failed: Msbuild failed when processing the file '[PATH]' with message: The imported project "[PATH]" was not found. Confirm that the path in the declaration is correct, and that the file exists on disk. [PATH]

27/09/2017 9:52:23 AM: First chance exception System.NullReferenceException: Object reference not set to an instance of an object. at Microsoft.CodeAnalysis.MSBuild.ProjectFile.Finalize()

27/09/2017 9:54:55 AM: First chance exception System.DllNotFoundException: Unable to load DLL 'Microsoft.VisualStudio.CodeMarkers.dll': The specified module could not be found. (Exception from HRESULT: 0x8007007E) at Microsoft.Internal.Performance.CodeMarkers.NativeMethods.ProductDllInitPerf(Int32 iApp) at Microsoft.Internal.Performance.CodeMarkers.InitPerformanceDll(Int32 iApp, String strRegRoot, RegistryView registryView)

I ran HtmlGenerator on TestSolution and there's no index.html either. SourceIndexServer runs fine from Visual Studio, but if I point IIS at the generated Index folder it still 404s on /.

KirillOsenkov commented 7 years ago

Hold on, if there's no index.html for TestSolution, how does SourceIndexServer run fine? Which index.html is it showing?

Are you able to build your project from the command-line msbuild.exe? Most likely, to build your project correctly, you need some environment variables to be set, and they're not being set in the HtmlGenerator environment.

Basically the root of your problems is in that Errors.txt. Get rid of those errors and things should work.

Also, is Errors.txt or Messages.txt being generated when you index TestSolution? Could you double-check that there's no index.html for TestSolution?

yaakov-h commented 7 years ago

I think that when I run SourceIndexServer from Visual Studio, it uses the wwwroot content from source code. When I run it from the generator output, there is no such folder.

I’ll check on the rest and report back.

yaakov-h commented 7 years ago

No Errors.txt, Messages.txt, or Index.html for TestSolution.

KirillOsenkov commented 7 years ago

@alexperovich hi Alex, we need your help.

After migrating to ASP.NET Core I don't understand the final layout of files on disk. What is the workflow of publishing a final website? How are the following parts laid out:

Dlls needed by ASP.NET Core (refs, runtime, root .dlls as well as web.config)
wwwroot files (index.html, scripts.js, styles.css, Content\Icons, etc)
Html content generated by HtmlGenerator

Where do all of these go, what's the final directory structure and what is the entire end-to-end process if someone wants to git clone SourceBrowser and publish their generated website to C:\FinalWebsite?

Thanks!

alexperovich commented 7 years ago

The workflow that I use for building source.dot.net is here The output executable from building HtmlGenerator.csproj should work as is. I added a custom build step here into HtmlGenerator that publishes the SourceIndexServer project into the output directory. The HtmlGenerator.exe file should have a Web folder next to it. This is copied to the output when you generate an index.

alexperovich commented 7 years ago

I just succeeded with the following workflow in powershell:

& "C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\15.0\Bin\MSBuild.exe" .\src\HtmlGenerator\HtmlGenerator.csproj "/t:Restore;Build" /p:SolutionDir=E:\SourceBrowser
E:\SourceBrowser\src\HtmlGenerator\bin\Debug\net46\HtmlGenerator.exe .\TestCode\TestSolution.sln
cd E:\SourceBrowser\src\HtmlGenerator\bin\Debug\net46\Index\
dotnet .\Microsoft.SourceBrowser.SourceIndexServer.dll

After that http://localhost:5000/ had a functioning index.

alexperovich commented 7 years ago

actually, after a git clean -xdff that workflow didn't work. It appears that the dotnet publish step for some reason isn't copying the wwwroot over correctly. You can work around this by copying the wwwroot folder from SourceIndexServer into the output directory that you passed to HtmlGenerator.exe

alexperovich commented 7 years ago

And for some reason removing the <EnableDefaultContentItems>false</EnableDefaultContentItems> from the SourceIndexServer project and the explicit listing of files fixes the problem.

alexperovich commented 7 years ago

It looks like this was a breaking change from 1.1 sdk to the 2.0 sdk. In the 1.1 sdk Content items were included in the publish output by default. In 2.0 Content items are only included in the publish output if CopyToPublishDirectory metadata is set on them. I will send a PR to fix this.

KirillOsenkov commented 7 years ago

Hold on, I'm confused. We don't have the CopyToPublishDirectory metadata on them, do we?

KirillOsenkov commented 7 years ago

@yaakov-h please try the latest and see if this fixes it.

KirillOsenkov commented 7 years ago

Hmm, thanks @alexperovich from my experimenting it works fine! However I don't understand how it is possible that the wwwroot folder is next to the index folder, but it all works seamlessly together?

I just run HtmlGenerator by pressing F5 and it generates a TestCode solution to SourceIndexServer\index, resulting in the following layout:

index
    index
    refs
    runtimes
    wwwroot
web.config
*.dll

KirillOsenkov commented 7 years ago

I'm guessing this is done here: https://github.com/KirillOsenkov/SourceBrowser/blob/master/src/SourceIndexServer/Startup.cs#L55-L59

Magic! Sourcery!

yaakov-h commented 7 years ago

I think that's because RootPath is set to the inner index folder here.

Thanks, I'll try again on Tuesday (AEDT) and let you know it goes.

KirillOsenkov commented 7 years ago

No, that line is only responsible for loading the list of symbols for the part that provides search results, the only non-static part of the site.

alexperovich commented 7 years ago

For asp .net core the default root path for the file server is wwwroot, so the plain UseStaticFiles() call serves content from wwwroot.

yaakov-h commented 7 years ago

Just re-ran with the latest HtmlGenerator. wwwroot is present and the site works.

Thanks!

KirillOsenkov / SourceBrowser

HTTP 404 on generated site #80